Fast Configuration Processing in SQL Race and ATLAS 10

Introduction

Fast Configuration Loading is an initiative to reduce recording startup and session loading times.

This is being done through four steps:

1. General optimisation of SQL Race and ATLAS 10 to reduce bottlenecks
2. A new, optimised format (FFC) that will coexist with PGV and CFG files
3. Re-engineered recorders, taking advantage of this format
4. Revised SQL Race API designed around the new format’s capabilities

Phase 1, available in time for Winter testing in 2019, covers most aspects of the first three steps, and lays the ground-work for step 4 in future.

Faster Recorder Startup

The recorder will still load PGV and CFG files, but the hand-off to ATLAS 10 should be more efficient when FFC files are available, reducing the recording start-up time substantially: expect start-up times averaging under 10 seconds. The first recording after starting ATLAS will still take longer (20-30 seconds from local disk) due to System Monitor folder processing, but the slow-down from having an empty recorder cache database has been largely eliminated.

Faster Session Loading

Sessions recorded with FFC should load from SSN2 or SQL Race databases into ATLAS 10 at least 50% faster than before.

A significant improvement should still be seen in both startup and loading even without the FFC files - but only in SQL Race: SSN files are unaffected.

Statistics

Performance will vary according to hardware and workbook, but these numbers may provide a useful guideline.

Timings were taken at the point where ATLAS is both responsive and scrolling data in the waveform from across all configuration and units, with a simple workbook. The hardware is a Dell XPS 15 laptop, with a local SQLite cache and PGV files on SSD.

Compared to previous ATLAS 10 releases:

• Recording startup time with an empty cache database is reduced by more than 80%
• Recording startup time with a warm cache database is reduced by 70%
• Historic session load time is reduced by 50%

Compared to ATLAS 9 releases:

• Recording startup time is reduced by 35%
• Session load time is reduced by 70% compared to SSN

In summary, this release should be faster than both ATLAS 9 and previous ATLAS 10 builds.

Measurements taken across a series of four sessions, starting with a clean session cache database and a cold ATLAS 10 client. The first three sessions include both TAG320 and HSL data.

Deployment

Generating FFC files from the System Monitor API

1. Install the latest release of System Monitor.
2. Download and review the attached Excel macro, which calls the System Monitor API.
3. Start System Monitor, and then run the macro to convert your existing PGVs.

Generating FFC files from the System Monitor user interface

1. In the Advanced Settings dialog in System Monitor, set the "Enable FFC Creation for PGV and Logging Config" option to "Y".
2. Use the Context menu for the App groups to create FFC files for PGVs on demand.
3. New logging configuration will include FFC content inside and alongside.

Update Scripts

Most teams have some scripted copying or caching of System Monitor folders.

We recommend extending these to include FFC files. Note that the recorder will still work correctly if some FFC files are not copied, falling back to conventional PGV processing.

Known Issue

It is advised not to enable FFC when programming vTAG, as there is an issue where it will fail to download RLC if the config is too large.

Performance Tips

Multiple-Core Parallel Processing

Previous versions of SQL Race loaded configuration on a single thread, but this new version will load configuration using multiple cores if available. This provides a significant part of the speed-up for historic session loading.

For best results - especially on Virtual Machines - take care to ensure that multiple cores are provisioned and that the hardware is not over-subscribed.

Metrics Log

We added a new ATLAS 10 log file late in the 2018 season, named ConfigurationProcessingMetrics.txt.

We would greatly appreciate your help in collecting performance statistics captured in this file, to help us measure and diagnose issues in production environments. Please help us to do this by collecting and passing these through to your TSE.

FAQ

How are FFC files created?

System Monitor will write FFC files automatically alongside PGV files, and as part of logging configuration.

You can export an FFC for a PGV from the System Monitor user interface, and there is a new API call for use from VBA, which is convenient for creating FFC files for existing PGVs. Logging configuration cannot be post-processed this way, however.

How should FFC files be handled?

Copy and store them together with the PGV and CFG files, as normal.

Are FFC files protected by RDA?

Yes. Most parts of the files are encrypted, and RDA rules are applied as normal.

How big are FFC files?

Approximately 75% as large as PGV files. CFG files will appear to be larger as they will embed FFC content, but System Monitor will avoid producing CFG files that are larger than the limit imposed by the ECU.

Will this require an ADS upgrade?

No, unless you are using Dual Recording into SQL Race, which will benefit.

Will this require a System Monitor upgrade?

Yes, but you can do a soft rollout by converting PGV files with the supplied script to generate the initial set of PGV+FFC files.

However, since logging configuration can only be FFC-enabled when it is written, you will need to update to the latest System Monitor release to get the full benefit.

Will this be available in ATLAS 9, or for SSN files?

No, due to technical limitations. Note that SSN files do not perform as well as SQL Race with ATLAS 10.

What’s the minimum database version?

If you are using SQL Server, the database needs to be at version 1.53 or later.

SQLite cache databases should automatically upgrade, but you may wish to delete and re-create them for a clean start.

When will this be available in production?

Winter Test 2019