Background

In this discussion, mention is made of a Model 204 monitor called SirMon. SirMon is written and distributed by Sirius Software Inc..

Model 204 is a high performance database package with an integrated teleprocessing monitor and programming language (User Language). While it is possible to do "batch" type processing with Model 204, its great strength is in the area of multi-user online applications. As with most multi-user applications Model 204 must control access to data structures that might be examined and modified by more than a single user. This control is known as concurrency control.

There are many different data structures that can be examined and modified by more one user.

File parameters

modified by CREATE, INCREASE, RESET and other statements examined by almost all file operations.

Field definitions

modified by DEFINE FIELD/REDEFINE FIELD examined when compiling any field reference in a User language procedure and when evaluating field references in IFAM programs.

Existence bit maps

modified by STORE RECORD, DELETE RECORD or FLOD examined by some FIND and FOR EACH RECORD statements.

Indexes

modified by STORE RECORD, ADD, CHANGE, DELETE, REDEFINE FIELD examined in FIND statements.

Table B pages

modified by STORE RECORD, ADD, CHANGE, DELETE examined in FOR RECORD loops and direct (Table B) searches.

Individual records

modified by STORE RECORD or FLOD examined in FOR RECORD loops.

The record locking table

examined and modified on all record locking operations.

The disk buffer pool

modified and examined in innumerable places.

Subsystem status

modified by START/STOP SUBSYSTEM, examined by $SUBSYS and MONITOR SUBSYS.

Procedures

modified by the Model 204 editor or the PROCEDURE statement, examined by the INCLUDE statement, APSY, the DISPLAY statement, $RDPROC.

The above is only a partial list of Model 204 data structures that are shared among multiple users. The Model 204 system manager or programmer will generally not need to worry about concurrency considerations for many of these resources. This discussion will focus on resources that tend to be associated with online performance problems.

The purpose of concurrency controls is to ensure that a user examining one or more data structures gets a consistent picture of the structures. That is, while a user is examining related data in these structures, it must be ensured that no other user is modifying the examined structure. For example, if a user compiles a program that contains a field reference, it must not be possible for the field to be deleted or for certain attributes to be changed (i.e. key/non-key) until that compiled program has finished running.

The general approach to providing concurrency control is by means of "locks". A lock is itself a data structure that can be examined and modified by multiple users. A lock is usually said to be "held" or "owned" by a particular user or users. Under the conventions of the programming environment certain kinds of operations on certain data structures cannot be performed unless the user holds a specific lock. Since a lock is a multi-user data structure, it is quite possible that obtaining or releasing a lock might itself require a lock since the act of obtaining or releasing the lock involves modifying the lock data structure.

Typically, a lock can be held in "share" mode or "exclusive" mode. When a lock is held in share mode, an unlimited number of users can also hold the lock in share mode but no users can hold the lock in exclusive mode. When a lock is held in exclusive mode, no other user can hold the lock in either share or exclusive mode. Typically, users need a lock in share mode to examine a data structure and in exclusive mode to modify the data structure.

Locks are associated with specific operations on specific data structures. They do not necessarily map one to one with either operations or data structures. A particular operation on a data structure might require several different locks to be held before it can be performed safely. A particular lock might protect many different operations on many different data structures.

The situation where a user is prevented from obtaining a lock by another user holding the lock is known as a conflict. There are three basic ways a conflict can be handled.

1. Wait indefinitely until the lock is available.
2. Return immediately to the user, indicating that the lock could not be obtained.
3. Wait for the lock to become available for a certain amount of time and return to the user after that time indicating that the lock could not be obtained.

Model 204 uses all three approaches with different locks.

Locks and performance

While locks make it possible to mix updating and retrieving applications in a multi-user environment, they have certain costs associated with them that have a negative effect on online performance. These costs are :

• The cost of obtaining and releasing the locks. This is usually a CPU cost under Model 204 except for record locks which can also require disk I/O. Usually, the cost of obtaining a lock is considerably greater when the lock is not immediately available than when it is.


• The waste of available resources while a user is waiting for a lock. For example, a system might sit idle while one user that holds a lock is waiting for I/O while another user is waiting for the lock. The user that is waiting for the lock might be using the idle CPU if the lock was available.


• Second order effects resulting from lock waits. A user waiting on a lock is generally still "consuming" certain resources. For example, a user waiting on a lock in 204 will occupy a server for at least a little while. The user might then get server swapped, requiring server I/O bandwidth. Pages the user has read in recently occupy space in the disk buffer pool. Eventually, these pages might require additional I/O to read them back in. In addition, the waiting user might be on queues that must be scanned by other users so there might be an second order CPU cost associated with a lock wait.

This not strictly fair strategy reduces user C's wait time by possibly increasing user B's wait time. In an extreme case, it is possible that if the lock continued to be held in share mode, user B would wait forever (or at least a very long time) before getting the lock. Thus a not strictly fair lock conflict resolution strategy can result in extremely poor response time for individual users even while overall system throughput is adequate. A strictly fair strategy strategy, on the other hand, will generate a more even response-time distribution at the cost of reduced overall system throughput.

Since in general, the cost of obtaining and releasing a lock when the lock is available is minimal (Model 204 record locks can be a notable exception) locks tend to have a significant negative effect on performance when there are lock conflicts, that is, when users have to wait for a lock. The probability of a lock conflict is roughly proportional to the average length of time the lock is held and to the frequency with which the lock is obtained. In the case of a lock that is always held in exclusive mode one can express this as

Reducing the frequency with which a lock is obtained can generally only be done by making changes to an application or to the nucleus code (Model 204). Reducing the length of time a lock is held, however, can often be done with system tuning, that is by setting system parameters or changing hardware.

In fact, since the length of time a lock is held is usually a product of the performance of one or more parts of a system, more often than not, what appears to be a problem with a lock is actually a symptom of some other bottleneck on a system resource.

Model 204 Locks

Model 204 has several basic categories of locks. Some of these are :

• User created locks (usually in work datasets)
• $function locks ($LOCK, $BIND)
• System resource locks
• Procedure locks
• Record locks
• File resource locks
• Critical file resource locks
• The CPU or "implied" lock
• MP locks

User created locks

can be used by User Language programmers to control access to global resources. For example, a programmer might wish to control access to external sequential datasets by means of a work dataset that contains records with the DDNAME's of the external datasets and the user number holding a lock on the DDNAME. It is difficult to make any generalizations about the performance characteristics of such a lock since this will be highly implementation dependent.

$function locks

are used by certain subsystems such as FILEMGMT, XREF and SIRMON. These types of locks are usually not heavily used and tend not to be a performance concern.

System resource locks

are used to control access to system wide data structures. For example, there are system wide locks that control starting and stopping of APSY subsystems, access to the password table (CCASTAT) and even one for the MONITOR DISKBUFF command. While any one of these can cause individual users to appear "hung", none of these resources are ever used by an evaluating User Language request. This means that they are not likely to be a performance concern for online User Language applications.

One exception to this is the system checkpoint lock. This lock is held in share mode by all users with active updating transactions in any file. The only user that ever gets this lock in exclusive mode is the checkpoint PST. Thus if the checkpoint PST is able to get the checkpoint lock in exclusive mode it can be certain that the checkpoint can be performed without any updating transactions active. This involves logging a journal entry and possibly rewinding the checkpoint dataset. The checkpoint PST will wait CPTO seconds for the exclusive checkpoint lock. While it is waiting, any user wishing to start an updating transaction will have to wait for either the checkpoint to complete or time out. Thus, a user with a long updating transaction can cause a chain reaction where it forces the checkpoint PST to wait for the exclusive checkpoint lock which then forces many other users to wait for the checkpoint PST. The checkpoint lock interacts with the file update resource lock in a way that will be discussed later.

Procedure locks

protect the integrity of procedures. These locks make it impossible for one user to modify a procedure while another user is examining the procedure. Procedure locks are never waited on, that is, they are the type of locks that indicate their unavailability to the user immediately. Because most User Language requests don't access procedures while they are evaluating, procedure locks are almost never a performance problem. There is a functionality problem that most shops have to contend with resulting from procedure locks. That is, APSY locks most procedures with precompiled and non-precompiled prefixes for the subsystem when the subsystem is started. This makes it impossible to modify these procedures while the associated subsystems are started. Incidentally, this lock is not intended to protect the procedures from modification but to protect in-core copies of the procedure dictionary entries for the procedures from modification.

Model 204 Record Locks

Model 204 record locks protect (as one might imagine) Model 204 records from modification by one user while another user is examining or modifying the same records. In general, the User Language programmer has complete control over how long records are locked by a particular request. FIND, FIND AND RESERVE and FIND WITHOUT LOCKS allow the User Language programmer to process records locked in share mode, locked in exclusive mode, or not locked. The one exception to this complete control is in updating a record. Whenever a User Language program updates a record, that record is locked regardless of the explicit locking requirements. The record is unlocked at the end of the the FOR RECORD loop or at the next COMMIT point

There can be no record locking conflicts if there is no record updating activity unless FIND AND RESERVE is inappropriately used. Thus, in read only database environments, record locks will not produce conflicts.

When a user attempts to lock a record or record set in share mode, all record numbers and record sets for the appropriate file that are locked in exclusive mode must be scanned for conflicts. When a user attempts to lock a record or record set in exclusive mode all record numbers and record sets for the appropriate file that are locked in either share or exclusive mode must be scanned. This is important for several reasons :

• If there are no updating transactions there should be no records locked in exclusive mode so that there will be virtually no cost to scanning empty exclusive lock chains. Thus, in read only database environments, record locks should not be a performance concern.


• There is a cost to holding a record lock for a long time even if it is not likely to cause conflicts since each held record lock increases the cost of any new record lock that might conflict with it, that is, in the same file, with a possibly conflicting mode (share or exclusive).


• Because scanning for conflicts might require I/O to read locked bitmap pages, a lock is required to control the scanning process. This lock is known as the RECENQ critical file resource.

Imprudent use of the RETRY statement in ON RECORD LOCKING CONFLICT units can produce deadlocks. If, for example, an ON RECORD LOCKING CONFLICT was coded as

Holding a record lock over a terminal I/O is generally considered a bad idea. Because the percentage of lock conflicts is roughly proportional to the average length of time a lock is held, and because terminal input is usually the slowest operation on most systems by at least a factor of 10 (usually over 100) holding a record lock over a screen I/O significantly increases the chance of conflicts. Even holding unlocked record sets (such as lists) over terminal I/O has a cost associated with it since all record sets, locked or otherwise, take up disk buffer pool pages and might result in CCATEMP I/O if held long enough.

Monitoring record locking conflicts presents special challenges. First, there is no statistic that counts the number of record lock conflicts. Second, when a user is waiting on a record lock, Model 204 presents its wait type as 7 which is the same wait type as most other lock waits. It is thus impossible to tell if a user is even waiting on a record lock. SirMon solves this problem by providing a statistic called WAITDSC (for wait description) that distinguishes record locks from other types of locks.

Another problem with monitoring record locks is that while it is possible for User Language code to determine the user number of the user causing the conflict ($RLCUSER), it is generally difficult for a system manager to determine the user causing the conflict.

When a record locking conflict occurs, Model 204 waits for the conflicting records to be unlocked. If the records are not unlocked within 3 seconds the user wakes up, re-finds if the conflict was on a find, and scans all locked records. This is done because it is possible that the record causing the conflict might no longer exist (due to a DELETE statement), might no longer appear in the found set (due to a CHANGE, ADD or DELETE statement), or might no longer be locked even though the found set is still locked.

If, after waking up, another lock conflict is encountered, the user is set to wake up when the new conflicting record set is freed or 3 seconds, whichever comes first. This is repeated ENQRETRY times. After ENQRETRY attempts the user is either sent to the ON RECORD LOCKING CONFLICT or ON FIND CONFLICT unit or given a "DO YOU REALLY WANT TO" message. If ENQRETRY is set to 0, any conflict immediately sends the user to the correct ON unit or DYRWT message. Thus if ENQRETRY is set to 10 and a user hits an unresolvable lock conflict, the user ends up waiting 10 times 3 or 30 seconds before a ON unit is even entered. That ON unit might have a PAUSE statement, do terminal I/O, journal I/O or even disk I/O before, perhaps, retrying the instruction that produced the lock conflict.

Thus monitoring a record locking conflict presents some unique problems. First, when a user is in a record lock wait, it will never appear that the user has been waiting more than 3 seconds for the lock even though the user might have already waited several 3 second cycles for a record lock. Second, if the user enters an ON unit as the result of a record locking conflict, and then issues a wait in the ON unit, there is nothing that indicates that the user is still trying to resolve a record locking conflict. Thus, even though to the end user there might appear to be a severe response time problem, to a monitor the picture might be much less clear. Understanding how record locking conflicts are handled does make it easier to interpret the information provided by a monitor, however.

Model 204 File Resource Locks

File resource locks tend to protect file structures against large scale file operations such as FLOD, Z, RESTORE, etc.. Presumably these do not usually cause any performance problems in an online system.

One notable exception to this rule is the lock on the so called UPDATE resource lock. The intent of the UPDATE resource lock is to synchronize the physical commit process (that is, the transfer of all updated pages from the buffer pool to disk) with other updating transactions. Specifically it is intended to prevent any transactions from modifying any pages in a file while that file is being physically committed. This ensures that at the end of a physical commit all pages on disk for a file are physically and logically consistent. If a transaction was allowed to modify a page for a file while it was being physically committed, the file might still be inconsistent at the end of a physical commit, since the page(s) on disk might contain a partially completed update transaction.

The UPDATE resource lock works as follows :

• When a user starts an updating transaction on a file it tries to get the UPDATE resource lock in share mode. The only reason this would fail is if a physical commit is being performed in which case the user performing the physical commit would have the UPDATE resource lock in exclusive mode. If this occurs, the user starting the updating transaction would wait until the physical commit is completed at which point the UPDATE resource lock would become available.


• During the updating transaction, the user holds the UPDATE resource lock in share mode.


• When the user issues either an explicit COMMIT statement or an implicit commit resulting from an end of request Model 204 checks if the user is the last holder of the UPDATE resource lock for the file. If not, the user simply releases the UPDATE resource lock. No physical commit is performed since the other holders of the UPDATE resource lock still have updating transactions in progress for the file.


• If the user is the last holder of the UPDATE resource lock for a file, the user upgrades his lock to an exclusive lock. This is guaranteed to work since this is only done when no other users hold the UPDATE resource lock.


• At this point the user writes out all modified pages for the file in the buffer pool. When this is completed the user releases the UPDATE resource lock.

The unfortunate side effects in an environment with many overlapping updating transactions are

• When a physical commit is finally performed, there might be hundreds of pages that require writing to disk. This could extend the physical commit process to several seconds holding up any updating transactions that start after the physical commit is started.


• Since updating transactions always hold the system checkpoint lock, checkpoints might be delayed indefinitely while these transactions complete.

Critical File Resource Locks

Critical file resource locks are essentially a variation on file resource locks. They are intended to protect critical (as the name implies) files structures from modification while a user is examining them or modifying them. They key differences between critical and other file resources are :

• Critical file resources locks can be obtained and released more efficiently.
• Critical file resources locks tend to be held for shorter periods of time.
• Critical file resources locks tend to be obtained and released with much greater frequency.

• The EXISTS resource lock protects access to the existence bit map.
• The INDEX resource lock protects access to the table C and ordered index.
• The DIRECT resource protects access to table B ("direct") records.
• The RECENQ resource protects scanning of record locking chains for a file. This is required because the record locking chains might contain bitmap pages which might require I/O to CCATEMP.

Except in an MP environment, the only way one can encounter critical file resource lock conflicts is if a user holding a critical file resource has to wait on something. If a user obtains a critical file resource lock, does some work and then releases the critical file resource lock without ever having to issue a wait, it will be impossible for another user to be dispatched by Model 204 while the resource lock is held. This means that is is impossible for a subsequent user to encounter a critical file resource lock conflict with the first user.

Typically, there are only 4 things a holder of a critical file will wait on :

• Disk I/O
• Journal I/O
• Checkpoint I/O
• Other critical file resource locks.

Based on this, it is clear that that any performance problem exhibiting a large number of critical file resource lock conflicts or long resolutions for these conflicts can be traced back to a journal I/O bottleneck, a checkpoint I/O bottleneck, high levels of disk I/O or slow service from the disk I/O subsystems.

To identify the root cause of such a performance problem it is necessary to identify what the holder's of critical file resource locks are themselves waiting on. Currently the only way of doing this is with SirMon.

The CPU (implied) Lock and MP Locks

Model 204 uses a technique called cooperative multi-processing to share a CPU among multiple users. This is different from pre-emptive multi-processing in that it ensures that the scheduler will never allow another user to run on the CPU until the currently running user voluntarily enters the Model 204 scheduler. While this has occasionally been the source of problems (such as tight CPU loops), the advantage of this technique is that it ensures that paths of Model 204 code are guaranteed to run uninterrupted by other users.

This provides a guarantee that certain manipulations of shared data structures can be performed without concern for concurrency issues. Thus, the fact that a user is running on the CPU is effectively a CPU lock.

With Model 204/MP, of course, a user can (almost) NEVER be certain that another user might not be running at same time. The concept of the implied CPU lock is still maintained in the MP environment, however, in that certain types of transactions are not allowed to be performed in an MP subtask, that is, they must be performed on the so called "maintask". All updating operations (ADD, CHANGE, DELETE, etc.) fall into this category. Access to the journal and checkpoint data sets is also only allowed on the maintask.

Access to three data structures that used to be protected by the implied CPU lock were considered so important however that forcing users of these data structure to run on the maintask would have rendered the MP feature useless. These three data structure are :

• The disk buffer pool
• The record locking table
• The Fortran library's non-reentrant work areas

• They are always held over relatively short periods of time (usually over a few hundred machine language instructions).
• A conflict never results in a Model 204 scheduler entry. Instead a conflict results in the task unable to get a lock issuing a real operating system (MVS) wait.
• MP locks are strictly fair.

Because of these features, users will never "hang" waiting for an MP lock. In fact, a Model 204 monitor will never actually detect a user in an MP lock wait. The main cost of lock conflicts, then, is the CPU cost of resolving the lock conflicts which is done via operating system WAIT's and POST's. This cost can, in fact, become quite significant in an environment where CPU usage is high enough to keep two or more tasks busy all the time. These locks are different from all other locks in the 204 environment in that the lock conflict rate completely measures the cost of these locks. The total number of operating system WAIT's and POST's issue for MP locks are provided by the Model 204 LKWAIT and LKPOST statistics, respectively.

User experience has shown that the disk buffer pool, or "DKBM" MP lock is the most significant source of lock conflicts. There are basically three ways that MP lock conflicts can be reduced :

1. Reduce usage of the disk buffer pool. This is likey to be difficult in most reasonably well written applications.


2. Try to force disk buffer pool intensive applications to run on the maintask, thus ensuring that they won't conflict with each other. Since this is a relatively new area, there are no documented user experiences with such an approach.


3. Bring in Model 204 internals experts (Sirius, CCA) to try to reduce the number of conflicts with changes to Model 204 nucleus code.

Conclusion

Model 204 concurrency controls are clearly not easy to understand. At the same time, conflicts resulting from these concurrency controls can both be symptomic of and exacerbate system performance problems. Because of this, two conclusions are inevitable for shops running complex multi-user onlines:

• It is well worth going to the trouble of learning more about how these resources work.


• It is essential to have a Model 204 monitor that allows you to easily and quickly monitor Model 204 lock conflicts. Currently, SirMon is the only monitor in this category.