IT audit course: Processing of IS Audit Evidence
Saturday, July 31st, 2010IS AUDIT EVIDENCE PROCEDURES
As has been stated, the auditor gathers evidence by following the audit program, which is a set of detailed steps that the auditor will follow in order to gain the appropriate evidence and, for the Information Systems (IS) Auditor, may well include the use of computerized techniques, although this is not always the case.
The evidence gathered permits the expression of an opinion on the efficiency, economy, and effectiveness of the activities. It lists directions for the examination and evaluation of information and provides the primary link between the audit field work and the audit report.
Steps in formulating the audit program involve determining the results from the preliminary survey, determining what, if any, risk is indicated; determining what types of controls best manage the risks; determining what, if any, additional evidence the auditor would like; and selecting the audit tests.
Like a route map, the audit program must fit the need of the traveller. It states what is to be done, when it is to be done, how it is to be done, who will do it, and how long it will take. The audit program helps the auditor stay on schedule/budget. It may be “pro-forma” or specifically tailored, but in either case it provides the following benefits:
- It is a systematic plan for each phase of audit work providing a basis for assigning work to the team members.
- It is a means of controlling and evaluating progress and assisting in training inexperienced staff members.
- It provides a summary record of work done.
- It reduces the direct supervision requirement by providing a clear path for subordinates and provides internal audit quality assurance information.
The final audit program should be prepared immediately after the preliminary survey although even then the program may be modified during the audit. Where new pro-forma audit programs are to be introduced, they should be prepared well in advance and field tested. Programs that are prepared too late will be rushed and have steps missing.
Preparation of the audit program should focus on what is hazardous to the corporation. The program should be thoughtful, relevant, effective, and economic, remembering again that not every item need be checked and that reasonableness and relevance should be maintained.
CRITERIA FOR SUCCESS
In order for the audit program to be successful, the objectives of the operation should be stated and agreed by the auditee up-front. The programs should be tailor-made where possible and the reasoning behind each work step should be shown. A common failing of audit programs is the creation of a list of questions to be answered. The audit program is a series of detailed instructions to be followed in the obtaining of audit evidence. These work steps should be prioritized to seek evidence of the most important control objectives first. All audit programs should be flexible because circumstances may change based upon evidence uncovered. Supervisory approval must be shown for all audit scheduling, staffing, and the audit constraints must be agreed.
Audit supervision will typically utilize standard project management techniques in order to match resources to requirements. This will include the defining, organizing, and monitoring of tasks and training staff as well as approval of the audit program.
The audit supervisor must be satisfied with the:
- Audit subject
- Audit objective
- Audit scope
- Pre-audit planning
- Selection of audit procedures
- Procedures for evaluation or testing
- Procedures for communication
- Report preparation
- Follow-up review
- The audit program provides for the collection of audit evidence of:
This may involve interviewing personnel, observing performance, or statistical sampling.
The overall audit approach must be:
- Simple
- Practical
- Quick
- Commonsense
- Business oriented
- Technically competent
STATISTICAL SAMPLING
In many cases the auditor can gain adequate assurance regarding the mitigation of risk without having to examine every single record or transaction. Under such circumstances, the auditor may choose to use a variety of sampling techniques in order to obtain evidence that is satisfactory and competent. The auditor may choose to use either non-statistical or statistical sampling techniques.
Non-statistical sampling involves the auditor making a judgment call as to the number of items to be selected and which items. The sampling technique is valid where the auditor wishes to examine a few examples without necessarily drawing conclusions about the whole population.
Statistical sampling itself is the process of testing a portion of a group of items to evaluate and draw conclusions about the population as a whole.
Statistical sampling may be defined as follows:
The auditor is performing either a compliance test, or a substantive test of either documented internal accounting controls or accounting source records by applying procedures to less than 100% of the items in the class of transactions or account balance for the purpose of forming a conclusion about some characteristic of the class or balance.
WHY SAMPLE?
The underlying assumption of sampling is that the results of a sample yield accurate information about the population from which the sample was taken. Sampling, therefore, can be viewed as an effective method of gathering audit evidence.
If auditors did not use sampling, every item comprising an account balance or every transaction occurring within a class of transaction would need to be reviewed. The cost of such an examination would (a) be prohibitive due to the amount of time required to perform such an examination and (b) far outweigh the benefit obtained. Sampling provides the auditor with a means of obtaining almost identical information, but at a much lower cost. Thus, sampling is also an efficient method of gathering information.
There are two basic sampling approaches:
- Judgmental/nonmathematical
- Statistical
Each approach represents a different way of handling audit risk. Therefore, each may be appropriate for some populations but not for others. Choosing the appropriate approach involves answering some critical questions about risk, population characteristics, and the objectives of our testing. The answers lead us to the best approach and the most efficient audit plan.
JUDGMENTAL (OR NON-STATISTICAL) SAMPLING
In judgmental sampling, the auditor relies solely on his/her professional judgment to assess the risk of sampling error and evaluate the population. Because the sample is not intended to be representative of the whole population, sample results cannot be extrapolated to the whole population. This approach is normally used where the auditor intends to use the sample for limited purposes.
Where the auditor is aware that a section of the population is higher risk, the auditor may choose to direct the sample to that particular area. Once again, the auditor has exercised professional judgment in selecting the population to be reviewed and conclusions drawn must be carefully judged to ensure their validity.
Judgmental sampling should not be used as a primary audit procedure if the auditors have no special knowledge about which items in the population are more likely to contain misstatements.
Again, judgmental sampling may be used for limited purposes (i.e., when sampling is not the primary audit procedure) such as corroboration of the outcome of other analyses by examining a few detailed transactions to check the validity of forecasts.
QUANTITATIVE METHODS
In addition to statistical analysis, a variety of quantitative methods are also available to be selected by the internal auditor. These mathematical tools are commonly used to obtain an understanding of operations and permit the drawing of conclusions in a variety of circumstances through analyzing the complexities of situations. Of the many quantitative methods available to the auditor, the following sections examine the most commonly used.
Trend Analysis
Trend analysis is used to evaluate the behavior of a variable such as turnover of a period of time. Such analyses can serve as evaluation criteria to determine the reasonableness of fluctuations of an extended period. Comparisons of this year’s turnover to last year’s turnover or, alternatively, this month’s turnover to the same month last year are popular.
Chi-Square Tests
Chi-square analyses are non-parametric tests capable of analyzing relationships between qualitative data. For example, do operating units in the South have particular patterns of operation different from those in the North?
Chi-square tests can check for the independence of normal classifications and ordinal data, and require no particular distributional pattern for the data.
Correlation Analysis
The measurement of the extent of association of one variable with another is known as correlation analysis. Two variables are said to be correlated when they move together in a detectable pattern. A direct correlation is said to exist when both variables increase or decrease in the same time although not necessarily by the same amount. For example, one would expect inventory to decrease as sales increased.
Correlation analysis is used by internal auditors to identify those factors that appeared to be related. An operational auditor, for example, may use correlation analysis to determine whether corporate performance is in line with industry standards by comparing the correlation of company costs of imported parts with the exchange rate fluctuations. Problems with how these statistics are computed; shortcomings in the internal auditor’s understanding of auditees’ operations, or real inefficiencies or misstatements can be pinpointed through correlation analysis.
Graphical Analysis
Graphical analysis can be useful to the internal auditor in identifying interrelationships in data, anomalies, and simple data errors.
A common form of graphical representation use by the auditor is a scatter diagram, which refers to any graph of data points. The more discernible a pattern appears in the graph, the more likely one variable is related to another and therefore can be used to predict the other’s value. Where no pattern can be noted, there would appear to be little, if any, correlation between the two variables.
Where a strong correlation exists, either positive or negative, the correlation value will approach 1. Where little correlation exists the correlation value will approach 0. Unfortunately, correlation values only measure linear patterns. Where there is a nonlinear relationship, correlation statistics will not disclose this. Occasionally the correlation value can be distorted by a single data point not conforming to the general pattern. While this can be readily seen on the graph, it may be less obvious in examining the correlation value.
Learning Curves
In conducting operational audits of performance levels of the implementation of new procedures for the quality of training of new staff, a learning curve would normally be expected to be observed. As employees gain experience with the new procedures or as the new employee becomes more experienced, the length of time taken to task should decrease.
Learning curves are evaluated by computing the time required per unit of production each time that the cumulative output is doubled. A decrease in production time per unit of 25% would result in a 75% curve. The 60% curve would result if the production time was reduced by 40%.
By measuring this curve the auditor can determine how quickly a new procedure or employee becomes productive. When a new procedure is recommended, calculating the initial time per unit under the old system and comparing it to a series of observations over time using the new procedures can objectively determine the impact of the revision to the procedures.
Ratio and Regression Analysis
Ratio analysis assumes a given proportional relationship between two numbers and is normally used for comparisons over time. A more advanced form of ratio analysis attempts to quantify the interrelationship in order to facilitate predictions in a regression analysis. Regression analysis is used to estimate the effect that a movement in one variable (the independent variable) causes a movement in the other variable (dependent variable); for example, if the sun shines, more cool drinks will be sold: but how many more? By performing the regression analysis the relationship, if any, can be identified and quantified and sales levels predicted.
Regression analysis can thus assist the auditor in understanding and quantifying data interrelationships. Unusual variations between expectations and recorded values may be noted for further investigation.
Using software, the auditor can additionally conduct a multiple discriminant regression analysis relating the independent variable to a number of dependent variables simultaneously. By determining the comparative strength of the relationships, the auditor can choose the focus area to achieve greatest impact in performance improvement. Such analysis has also been used to attempt to predict bankruptcy.
As with most statistical tools, regression analysis is based on a set of underlying assumptions that must be met for its use and interpretations to be valid.
Linear Programming
Linear programming is an operations research tool used for the allocation of scarce resources or to determine optimal blends of raw materials. The constraints applicable are reduced to algebraic formulae, which are then solved by simultaneous equations. For example, in a production environment, machining may be capable of processing 100 units per machine, while finishing can handle 35 units per machine. The question of how many of each machine should be used for optimum production can be solved using linear programming.
PROJECT SCHEDULING TECHNIQUES
Accurate project scheduling techniques have long been a goal in project management. Internal auditing frequently works in project teams that often suffer from the same poor project scheduling.
Program Evaluation Review Techniques
The program evaluation review technique (PERT) is used to diagrammatically identify dependent and independent activities. By showing graphically which activities cannot be started until the previous activities have been completed and, at the same time, which activities can proceed simultaneously, the planner can allocate resources to those tasks having the most impact on the final completion deadline. This technique also takes into consideration operational constraints placed on the resources needed to carry out the tasks.
In Exhibit 9.2, the shortest time to get from A to E while completing all tasks is calculated by calculating the longest path. Other times and paths include:
- Path A-B-C-D-E takes 8 days
- Path A-F-G-D-E takes 5 days
- Path A-H-I-E takes 9 days
EXHIBIT 9.2 PERT Chart
This means that the bottom path would be the most critical. The reason for this is that any delay in this path will postpone the final completion date. Any delay in the middle path that does not exceed four days will have no effect on the final completion date. Should the top path experience a delay in any of the processes of, for example, three days, then the top path will now take 11 days to complete and will become the critical path. If, by the same token, the time taken for the critical path can be reduced, then final completion date can be brought forward.
Critical Path Method
The critical path method (CPM) is a scheduling tool that was developed independently of PERT but uses a similar diagram. CPM, however, uses two time estimates: one for normal effort and one for “crash” effort. “Crash” time is the time required for completion if all available resources were committed to that task.
GANTT or Bar Charts
One of the simplest planning tools requiring no mathematical calculations is the Gantt chart. It is commonly used in organizing work and monitoring progress through the various stages of a simple project and involves the production of bar charts showing the start and completion times of individual project activities. The major drawback to these charts is the poorer representation of interdependencies.
SIMULATIONS
Monte Carlo Simulations
Computers can be used to accelerate timescales by carrying out activities repeatedly very rapidly. By combining this with the probability of events occurring, a sophisticated model can be built.
One such approach is referred to as the Monte Carlo Method. It uses the computer to simulate uncertainty via random behavior based on the probabilities entered and then estimates specified models several times to determine average performance.
Game Theory
The term “game theory” refers to mathematical models of optimal strategies under various incentive schemes. This is used in competitive environments to explore “what if” solutions.
A non-zero-sum game is said to exist when a profit is generated in which it is possible for both participants to share. A zero-sum game denotes a situation where a profit simply transfers from a loser to a winner. Game theory is used to assist the internal auditor in understanding the reasons particular strategies are pursued in negotiation sessions or competitive price setting.
Queuing Theory
Businesses often have queues at service points. Elimination of these queues by increasing the number of service points would result in service points frequently being unused and costs increasing. Management frequently must decide how many service points should be provided.
Queuing theory facilitates the use of mathematical models to minimize the total cost for a given rate of arrivals; the minimized cost includes both service costs (facility and operating costs) and waiting costs (the idle resources waiting in line or having service points idle).
COMPUTER ASSISTED AUDIT SOLUTIONS
In today’s environment, a review of business systems will almost inevitably involve the use of appropriate information retrieval and analysis programs and procedures. The auditor will use test transaction techniques to review system-level activity. In advanced auditing, the use of knowledge-based systems will permit the distribution of advanced audit techniques to less skilled staff. Confusingly, these are commonly referred to as Computer Assisted Audit Tools, Computer Assisted Audit Techniques or more correctly, Computer Assisted Audit Tools and Techniques.
Computer Assisted Audit Tools and Techniques are needed because of the large volumes of data in multiple locations involved in the managing of a complex business environment.
The use of computer assisted audit solutions involves the merging of software into an audit program. In order for this to prove effective, key control questions must be predefined in order to facilitate the use of the technology to analyze the data and provide the answers.
Advantages from the auditor’s perspective include increased auditor productivity, creativity, and the application of a consistent methodology.
Information retrieval and analysis programs and procedures include programs that organize, combine, extract, and analyze information. This includes generalized audit software as well as application and industry-related software. Customized audit software and information retrieval software as well as standard utilities and on-line inquiry may also be utilized for information retrieval and analysis. Where the auditor has computer skills in programming, conventional programming languages may provide a viable alternative, but a lack of such skills does not preclude auditors from utilizing such techniques. The ready availability of microcomputer-based software, which provides computing power without the requirement of technical expertise, puts direct data analysis within the toolkit of any auditor. The primary requirement is an understanding of the business application and how data relates.