Simulation speed

© Copyright 2018 Energy Systems Research Unit, Glasgow, Scotland

Those who use simulation tools are familiar with waiting for the kettle to boil. Unlike the impatient cook, simulation users can actually influence the resources required to carry out tasks -- if we understand the resource implications in how we design our simulation models, the workflows we use and the bottlenecks within our computing platforms. Improvements of an order of magnitude in specific tasks are possible. The method used to highlight these opportunities is the creation of a test matrix which includes simulation models at various levels of complexity, various assessment goals, various simulation frequencies and performance-data-to-save directives, computational rigour directives across a mix of legacy and current computing platforms, operating systems. And just to spice it up we will look at the implications of the tool chains used to build simulation tools and run alternative simulation tools across the matrix.

What we found:

User choice implications

What we choose to include in our virtual worlds of physics and our design decisions within those virtual worlds have consequences. Although simulation tools may guide (or coerce) us, decisions about the extent of our models and the levels of detail it contains is ultimately down to the model design decisions of practitioners. Following the easy path though the tool does not necessarily create a model which is fit-for-purpose. Enough said. The benchmarking matrix includes two models. The first is a pair of cellular offices with a corridor. Its composition and complexity characterises the exemplar training models distributed with simulation suites such as ESP-r and EnergyPlus. The second model was taken from a project on retrofit options for traditional heavy masonry apartments.

A simple training model

Models used for training in simulation tools tend to be constrained in complexity. Often training models have a half dozen zones, perhaps 40-50 surfaces and straightforward environmental controls. Such model constraints reflect novices abilities and, in general, assessment tasks associated with them make minimal demands on computers.

The figure to the left is a training model distributed with ESP-r. It has two identical cellular offices with a corridor. Glazing frames are separate surfaces and some of the walls have been subdivided. This model was also exported as an EnergyPlus IDF file V8.8 so the assessment timings are well matched in terms of the zones, surfaces, surface composition, casual gains as well as startup days, assessment periods and timesteps. However, to accommodate the Conduction Finite Difference solver in EnergyPlus (see grumble later on), the conductivity of steel in the floor structure was reduced and a fibre concret rainscreen was substituted for the standard metallic rainscreen in the facade. ESP-r uses a finite volume solver which is less sensitive to high conductivity materials (complaints start with layers of 0.4mm thickness). The contents of this model are reported in this ASCII contents file or as a PDF file

The periods of assessment, timesteps, warmup periods and solar shading directives for the EnergyPlus model were also matched. Both 4 TSPH and 20 TSPH and both the CTF and CFD solvers are exercised in EnergyPlus and ESP-r uses its Finite Volume approach. One difference is the use of a free floating control for the EnergyPlus runs.

High-mass traditional apartment

Models which represent real buildings often explicitly represent most, if not all, of the rooms in the building as well as specific aspects of the building facade. They may also abstractly represent adjacent spaces which represent bounding conditions and they may also include a degree of diversity in the occupancy and casual gains within the spaces so as to capture the response characteristics of the building.

To assist our comparison with EnergyPlus 4 & 20 TSPH assessments have been run. Twenty TSPH yields an annual results file of over 21GB (full zone and surface energy balances). Although the ESP-r res module can handle this the pedantic nature of the post processing requests tends to thrash the disk, indeed the quantity of data written also impacts the speed of the simulation. As will be discussed in a later section, it is possible to request that performance data be saved at a lesser frequency and to write a subset of data to the file. The latter technique has been used for the summer and annual 20 TSPH runs performance data was saved once per hour than at each timestep. User choices for recording performance data which impact on the size of the files are also explored. In a later section we will look at the implications across a range of user directives.

As with the simple office model the traditional apartment was exported to EnergyPlus version 8.8. Give the mass of the structure the Conduction Finite Difference solver is the solver of choice (the CTF solver is also tested). In this model only one instance of steel needed its conductivity relaxed to avoid a severe error. Again the assessment was run with a free floating control.

For this study we will use an 1890s apartment in a sandstone building. Such models often represent each room explicitly as well as the bay windows in the facade and the different thicknesses of stone (up to 900mm thick). In this case there are 13 zones and 432 surfaces. Environmental controls involve radiators (mixed convection & radiation) with TRV (mixed sensors). The figure below shows a view (including an abstract bounding zone below the occupied space as well as the roof space above). The usual range of timesteps for this model is 8-10 TSPH (there is nothing in the model or the goals of the project which demands a shorter timestep). The contents of this model are reported in this ASCII contents file or as a PDF file

Directives on simulation frequency

Our choices on assessment timestep is typically a compromise between the response characteristics of the building and its environmental controls, the what-do-we-need-to-record in order to confirm the design works and the constraints of the solver being used. In the matrix we use both a 15 minute timestep and a 3 minute timestep. Both tools support a minute timestep for the building solver and ESP-r can simulate plant at a fraction of a second although the latter is not explored in this study.

Ideally the timestep reflects either the frequency at which we want to explore performance issues or an aspect of an environmental control or mass flow which might undergo rapid changes.

Some solvers constrain the simulation timesteps - in the tables below the CTF method in EnergyPlus does not like frequencies less than 4 per hour (not sure about frequencies greater than 30 per hour). Most example IDF files use 4 or 6 timesteps per hour. The Finite difference method does not allow frequencies less than 20 per hour (20 30 or 60 are possible).

The extent and frequency of performance data captured during an assessment forms another matrix to explore. ESP-r has a concept of save levels two of which are explored - a full set of performance data (i.e. full energy balances at zones and surfaces) and a subset of performance data. The tests include both save levels and for longer periods a data frequency of hourly as well as each timestep. Post-processing in ESp-r recovers the following performance data for each zone:

The EPP assessment writes hourly data on:

as csv and ESO files but no further post-processing. These directives would need to be considerably extended to be comparable with what ESP-r stores. There is also no direct equivalent of the ESP-r res module so the method for comparing post-processing timings needs to be evolved.

To test the impact of writing data at higher frequencies (see disk-bound section below) variant IDF files were created with timestep directives.

Directives on simulation periods

To highlight the costs of uncritical generation of annual performance indicators and focused reality checks to gain confidence in models we include in the matrix the following assessment periods:

Pre-processing directives matrix

Directives which impose additional resolution on models, for example, computing surface-to-surface viewfactors, shading & insolation directives are computed prior to assessments and the stored results used during assessment.

Simulation tools offer a range of user directives for radiation exchange within zones. The process of computing and using surface-to-surface view factors in ESP-r is straightforward and seems to be where dragons live for EnergyPlus. In ESP-r the method is based on creating a grid of points on all surfaces. The grid resolution defaults to 10x10 per surface but can be increased (e.g. to cope with thin frames around doors and windows). Shifting to a ~40x40 grid would impact computing resources but is rarely required.

For ESP-r timings were taken on a Dell 7010 running Linux and ESP-r V12.7. The task of pre-computing surface-to-surface view factors rarely takes more than a minute and is often a matter of seconds. Pre-calculating the hourly distribution of solar insolation within a complex zone (including to the explicit mass surfaces) requires even fewer resources.

Two levels of romm complexity have been tested. The room on the left includes 314 vertices, 152 surfaces of varying complexity from 4-36 edges in the facade as well as dozens of explicit internal mass surface pairs representing desks and filing cabinets. Also seen are the location of MRT sensors (at occupant head locations for assessing radiant asymmetry).

The pre-processing in the context of a simple zone (90 vertices 40 surfaces) such as that found in the simple pair of offices implies the following overhead:

In addition to the time taken for actual calculations the specification of user directives for shading and insolation requires a few seconds. There are a few options to set in terms of the accuracy of viewfactor calculations as well as locating and defining MRT sensor locations.

As zones evolve shading is automatically updated. If the location on site is adjusted all shading is automatically updated and because there is no user interaction needed it may take only a few seconds per zone. Viewfactor calculations require the user to initiate the calculations. In a 90 zone model this becomes a repetitive task that probably should be automated.

Assessment directives matrix

Simulation models include a range of directives which impact computing resources.

Hardware choices

Each computing platform includes a mix of hardware which can impact how much time the kettle takes to boil. Each epoch of computer has quirks as well as opportunities for fine tuning. And there are the disruptor platforms which may find application in some simulation scenarios.

Computer matrix

The following table describes the matrix of computers included. The columns are:

Timings for a simple training model

In this table the assessment times are shown for ESP-r V12.7 compiled with no optimisation and high optimisation with the GCC tool chain on all computing platforms. Where EPP V8.8 was locally compiled the same compiler and optimisation levels were used. For Windows 10 the EPP suite was a pre-compiled download.

This particular ESP-r model is normally run at 4 TSPH, however the CFD solver in EnergyPlus requires at least 20 TSPH so ESP-r runs were made at both 4 and 20 timesteps per hour.

In the table below times are in seconds use the syntax for ESP-r is (4tsph simulation/reporting) [20tsph simulation/reporting] and for EnergyPlus is (CTF conduction transfer function simulation & ReadVars) [CFD conduction finite difference & ReadVars].

Discussion

The table indicates that reality-check runs are in the order of seconds and seasonal/annual assessment tasks of one minute or less across most of the matrix of computing platforms. Unless you are undertaking automated parameter excursions the interactive user experience is little changed across the matrix.

At this scale there is no need to use the legacy EnergyPlus CTF solver (unless you are running on one of the single board computers or running longer assessments on legacy kit).

Windows shows a disadvantage for disk intensive activities. The EnergyPlus downloaded executables on Windows have an advantage over locally compiled Linux versions.

There appears to be little difference between the performance of non-commercial and commercial virtual computers for simple models. Indeed, at this scale of model there is little or no increase in assessment times in comparison with native tool performance (even with constrained memory).

For summer and annual assessments the use of un-optimized simulation tools shows a significant increase in timings. Essentially one would avoid un-optimized builds for production work, especially with models with any level of complexity.

Timings for the High-mass traditional apartment

In the table below times are in seconds use the syntax for ESP-r is (4tsph simulation/reporting) [20tsph simulation/reporting] and for EnergyPlus is (CTF conduction transfer function simulation & ReadVars) [CFD conduction finite difference & ReadVars]. A * indicates performance data generated at 20 TSPH but saved hourly.

Discussion

As the complexity of models increases different patterns emerge in the interplay between user directives and hardware.

Disk-bound assessments

Writing performance data and recovering it for inclusion in reports and graphs might be another elephant-in-the-room. The tests suggest a number of correlations with:

As seen in the above tables, simulation tasks via models such as the simple pair-of-offices are rarely disk-bound. The slow SDHC disks of some single board computers are somewhat impacted but with most legacy and current computers in the matrix users will hardly notice.

The table below illustrates the time implications in the context of the traditional apartment:

As models approach the complexity of the traditional apartment instances of disk-bound assessment begin to be noticeable, in some cases they begin to dominate. Where data recovery agent requests are supplied via the memory buffer tasks can be accomplished one or two magnitudes quicker. What we notice is that for disk writing from the simulation engine in ESP-r is roughly 10-50MB/second while disk reads, especially from large data stores might be half that speed. Where post-processing can take advantage of the operating system file buffers the data recovery is one or two orders of magnitude faster. Remember, writing of performance data is essentially a sequential task while post-processing of multiple topics can involve random access into the performance access if not multiple passes over the data store to derive statistics and/or generate graphs.

Although it does not show up in the graph, response to user interactions for non-simulation tasks are impacted, especially during post-processing. Pointers would freeze and simple tasks like folder listings could be an order of magnitude slower. Multiple processors may speed automated tasks but the user experience is degraded.

Users have a number of choices:

Workflows

The ordering of simulation tasks can have a substantial impact. Assessments which generate a large performance data sets and which are immediately followed by data recovery actions tend to be more efficient than running a series of assessments followed by a sequence of data recovery tasks.

On most computing platforms, even those with multiple processors, users will notice sluggish response for tasks carried out at the same time as heavy disk activity. To explore this consider the table below which looks at the time required for sequential and parallel simulation tasks.

On workstations with a reasonable hardware provisions it is possible to speed up simulation tasks by invoking assessments in parallel. Four cores never runs four assessments in the same time as a single assessment. In the table above the monthly assessments each include a warm-up period so the sum-of-sequential is greater than a single run covering the season. The parallel runs does save time - but of post-processing needs to span the whole season then the bookkeepping to aggregate predictions may be problematic.

On computers which have multiple cores but are constrained by memory and/or disk I/O the story is mixed. The table shows that post-processing dominates the seasonal run. The parallel assessments do run in a fraction of time but parallel post-processing is disk-bound.

Is this pattern unique to ESP-r? The same scenario was run for EnergyPlus V8.8 on the same computing platform. The results are presented below for the Conduction Finite Difference solver at 20 TSPH:

The patterns are, indeed, similar. A summer assessment is somewhat more efficient than sequential runs but it takes roughly half the time when running 4 assessments in parallel. EnergyPlus is doing much less post-processing so no clear pattern emerges.

Observations

More often than not we select our matrix of model attributes, assessment directives and mix of hardware based on habit rather than using benchmarks. This risks over-and-under-specified kit for the goals of projects. For example, a high resolution monitor could be much more important to model creation than cutting edge processing capabilities. If we anticipate disk intensive tasks will be a bottleneck then fitting additional memory and an SSD can often result in better turn-around than a faster processor.

The research indicates that the timings for EnergyPlus's Conduction Finite Difference solver are less dire than is usually assumed in the community. Yes, it is a hassle that the conductivity of some metal layers need to be relaxed to use the Finite Difference solver (such sensitivities only kick in for sub-millimeter layers in ESP-r). The 20, 30 or 60 time steps per hour requirement is more restrictive than in the ESP-r solver. Surely the EnergyPlus development team could sort the metal layer conductivity limits and find ways to support somewhat longer timesteps. Both of these would encourage a default use of Finite Difference solver. Experts should, or course, be able to choose CTF.

Although ESP-r's res module is very efficient when the data store is less than the available memory buffer or where the data store is less than ~2GB post-processing tasks are hardly an issue. The 21GB data store highlights inefficiencies in the logic which controls random access to the file store. The process is disk bound more by the speed of the read statements and multiple passes through the data store rather than the raw speed of the disk drive.

Meltdown and spector

In January 2018 faults in a range of CPU architectures were uncovered with speculation that the required patches might slow down some user tasks. The table below explores the implications on simulation tool timings for a subset of the test matrix. The 1890s apartment building has been used and the following notes explain the hardware and software context:

In the table below times are in seconds. The syntax for ESP-r is (4tsph simulation/reporting) [20tsph simulation/reporting] and for EnergyPlus is (CTF conduction transfer function simulation & ReadVars) [CFD conduction finite difference & ReadVars]. A * indicates performance data generated at 20 TSPH but saved hourly.

Discussion

The scale of performance change seems to vary depending on the mix of operating system and simulation tool. Some changes are small enough to be attributable to noise in the testing (i.e. other background OS tasks). ESP-r is impacted more than EnergyPlus for simulation phases as well as for post-processing.

Next steps

There are gaps in the benchmarks: