meta data for this page
JEA DLL for DesignBuilder
This page provides the details of the JEA API of the local DLL for DesignBuilder. A demo app is included in the package to demonstrate the working of the COM interface and the JEA engine.
For more technical details of the JEA Engine and the JEA Web online service, please see: API for the JEA Engine
This document is last updated on 3 June 2019. The latest DLL can be found in the attachment ofthe email sent on 03/06/2019.
Summary of changes
Latest changes 2019-06-03
- Version number: updated to V1.0.2 Stable 1. Version number of the engine is displayed in the TRACE log at the beginning of each run.
- JEA core: termination criteria for number of evaluations, cpu time, and wall time are implemented. It also checks if the effective search space has been explored, and terminates if true.
- JEA core: for optimisation, all sampling methods are supported for generating the initial population.
- JEA core: descriptive and regression stats are always produced after each run, should it be terminated naturally or cancelled by the user. A global stats record on all solutions is also kept with reference '-99'.
- Data structure and API: the enums in project configuration such as the algorithm and the sampling method options are now case insensitive.
- Data structure and API: EAResultsArchive.listAnalysis are now indexed by the run number instead of the generation number. The global stats record has the special index -99.
- Bug fixes: P-value for intercept in regression analyses were incorrectly calculated and is now fixed.
Latest changes 2018-02-24
- JEA core: if empty results are returned in job submission, those cases are considered to have simulation failures. The cases are ranked worst in the population and marked as
Valid == false
in the data archive. They are excluded from all statistics. - JEA core: in regression analysis, added regression parameter Standard Error and P-value, and regression residual.
- Data structure and API: each JSON object in EAResultsArchive.solutions has an additional field
valid
. - Data structure and API: EAResultsArchive.listAnalysis has been changed. Regression analysis results are now stored in their own map named
regressions
. - Data structure and API: stats can now be called upon by including a
filter
object (can be empty) in the Report command.
Latest changes 2017-12-17
- JEA core update with bug fixes, including Saltelli sample size changed to N x (D+2)
- Hooked up with the JEA Web debug server (
jess.ensims.com:2997/jea_web/api/
) with DB debug key and user (debug@designbuilder.co.uk
) - Configuration file for JEA DLL initialisation
- Additional API functions for changing JEA DLL configs and retrieve project list and data
Changes 2017-08-24
- Added truncation, scaling and shifting to supported PDFs
- Internal representation of the numerical values changed to BigDecimal
- Various bugs were found and fixed in the JEA engine
Changes 2017-01-10
- Added probabilistic distribution definitions for parameters. Supported PDF functions include:
- Integer: Bernoulli(1), Binomial(3), Geometric(9), Hypergeometric(10), Pascal(NegativeBinomial, 13), Poisson(16), DiscreteUniform(20)
- Continuous: Beta(2), Cauchy(4), pdtChiSquared(5), Exponential(6), F(7), Gamma(8), Levy(11), LogNormal(12), Normal(14), Pareto(15), T(17), Triangular(18), ContinuousUniform(19), Weibull(21)
- Added sampling methods:
- Pseudo random
- LHS
- Sobol
- Halton
- Random Walk (one parameter at a time, single step, N x (d + 1))
- Morris (one parameter at a time, random step size, N x (d + 1))
- Saltelli
(Sobol N x (2d + 2))(Sobol N x (d + 2))
- Implemented Morris method, to be applied automatically on Random Walk or Morris samples
- Implememted Sobol method, to be applied automatically on Saltelli samples.
(Issues remain with the library used) - Implemented Regression analysis (using ordinary multivariate linear regression). Regression parameters and standardized coefficients (sensitivity measure) are calculated for every output variables. Applied automatically on each complete optimisation, parametric or sampling runs. Or calculated on report request of partial runs.
- Regression results, descriptive statistics, and frequency histogram are reported in the new Report transaction
- Full project data accessible using
the new Data transactiongetProjectData()
API function - Added Info transaction to retrieve engine version information
- Update command fully functional
- Added Reset command to clear all data in the current project
- Other changes:
- Transaction/command data:
- Model version/reference can be specified with Create/Update commands
- Computing time can be submitted with simulation results
- Additional fields in EA configuration object
- JEA internal logic:
- All interactive features implemented and partially tested
- Stop-start logic changes, see Engine States Diagram
- Extension to project data archive, see Project Archive
Changes 2016-10-22
- Separating engine response and engine status packets. An engine response packet is returned immediately to a command. It contains only a 'ok' and a 'description' field. An engine status packet is returned when status is enquired (with a status command). It contains 'projectName', 'status', 'generations', 'solutions' fields in addition to 'ok' and 'descritpion'.
- Canceled is added to the engine status codes, which now include:
- Unknown,
- Created,
- Initialzing,
- Evolving,
- Paused,
- Terminated,
- Canceled,
- Error
CancelRequest
uses the same Callback delegate as 'Request', whereas 'StatusChanged' and 'Logging' require their own callback delegates respectively. API functions for setting the delegates have changed. See the Set Callback Delegate section for details.- API functions for retrieving event messages have been changed. To read Request, CancelRequest and StatusChanged event messages, the project ID must be specified.
- The TestGUI app has been updated to demonstrate the latest API and events
- Improvement to this document
Package description
This developmental package is based on Graham Povey's proof of concept implementation of the COM component and the Java interface. (Graham's original notes and download link are available.) It contains the projects of the COM component, a VB.NET GUI for demonstrating the use of the API, the JEA DLL and its compiler, and the working directory including example files and a utility app for viewing optimisation history.
Contents
The package includes three folders in addition to Graham's original four. The additional folders are:
resources\JavaSource\
- contains the latest Java code for the jea_ikvm_wrapper. The source code is for information only. The library has to be built in an IDE with other dependent projects, which are not included in the package. Ready-built .jar and .dll files of the Java library are available in theIKVM Java
folder, namedjea_ikvm_wrapper-1.0.0.jar
andjea_ikvm.dll
, respectively.run
- this is the folder containing the executable and libraries, the example problems, the configuration files, and the outputs of the demo. Two example optimisation problems, BINH&KORN.json and CEC2009_CF2.json are included for testing using the demo app. The optimisation project data files are all located underlocal/
. Logs and errors of the Java library is stored indb-jea.err
file, whose name and log level can be defined inlog4j.cfg
. An important new addition in this release if thejea_config.json
, of which the details are provided here.TesterGUI
- contains a VB .NET GUI for testing the coupling mechanism and the Java library. It is written in .NET due to the lack of VB6 IDE. The GUI project is loaded as part of thevb6EventClassLib
solution in theCOM Wrapper
folder.
Running the demo
I am not sure if the COM component and the DLLs need to be registered before launching the demo or not. If they do, it may be easier to start the app from VisualStudio. Load the solution in COM Wrapper/
to load the projects, and then click on start.
The defaults of the JEA DLL has been changed to local, and a project name is no longer required for initialising the COM dll. Switching between online and local JEA and setting other configurations can be done either with the initialization configure file, or with the new API functions when the DLL is loaded.
... public Vb6Event() { DBEAServiceInterface.setCallback(SimRequestCallback); DBEAServiceInterface.setStatusChangedCallback(StatusChangedCallback); DBEAServiceInterface.setLoggingCallback(LoggingCallback); DBEAServiceInterface.setUseJEAOnline(false); } ...
If started ok, the window below will show:
The sequence to operate is:
- Select the example problem to run
- Create EA
- Send Command
- Start
- Send Command
- Dismiss status dialog box
- Dismiss new jobs dialog box
- Calculate result
- Send Result
- … continue
- Pause, Resume, Terminate, Cancel, Status and Report etc. followed by Send Command
- Get Project List and Data
- Remember to change project name (in the commands to be sent) or delete the existing project to reuse name
The activities will be shown in the console output and stored in db-jea.err
. All projects and the project data are stored in local/
.
Configuring the JEA DLL
A new configuration file which is loaded at the initialization of the JEA DLL with default settings is introduced in the 2017-12-17 version. Most of the configuration parameters can be set through the API functions (see the next section), too. Here is a configuration file as included in the package:
{ "rootFolder" : ".\\", "logConfigFile" : "log4j.cfg", "projectsFileName" : "projects.json", "useJEAOnline" : false, "jeaonlineUrl" : "http://jess.ensims.com:2997/jea_web/api/", "jeaonlineApiKey" : "117d18e...08f85fb8faa", "jeaonlineUser" : "debug@designbuilder.co.uk", "purgeOnStart" : false }
Most of the parameters are self-explanatory. Please note that the JEA's error log file name is defined in the log configuration file (log4j.cfg
). Its location is overridden by the routFolder
parameter.
JEA_ikvm Module API
// ===== Main operations ===== /** * Get Engine info * @return Engine version info in JSON */ public static String getEngineInfo (); /** * Execute the JSON command * @param jsoncmd JSON encoded DefEACommand object * @return DefEngineResponse object */ public static String execCommand (String jsoncmd); /** * Synchronously set simulation results and return EA engine response * @param results Results wrapped in a DefSimulationReturn object, as JSON string * @return Engine response as JSON string */ public static String setEvaluationResult(String results); /** * Get the list of projects from the engine * @return An array of project names */ public static String [] getProjectList (); /** * Get the archive data of the specified project * @param prj_id Project name * @return JSON representation of the project data archive */ public static String getProjectData (String prj_id); // ===== Setting up callback ===== /** * Set the callback delegate for simulation request and cancel request events. * A delegate must be set during initial creation of JEA * @param Callback The callback delegate object */ public static void setCallback(RunnableDelegate Callback); /** * Set the callback delegate for Status Changed events. * A delegate must be set during initial creation of JEA * @param Callback The callback delegate object */ public static void setStatusChangedCallback(RunnableDelegate Callback); /** * Set the callback delegate for logging events. * A delegate must be set during initial creation of JEA * @param Callback The callback delegate object */ public static void setLoggingCallback(RunnableDelegate Callback); // ===== Retrieving callback message ===== /** * Retrieve the callback message set by simulation request and cancel request * events. A project name is required to identify the correct engine * @param prj_id The ID string of the project * @return Stored callback message */ public static String getCallbackMessage(String prj_id); /** * Retrieve the callback message set by status changed events. A project name * is required to identify the correct engine * @param prj_id The ID string of the project * @return Stored callback message */ public static String getStatusChangedMessage(String prj_id); /** * Retrieve the callback message set by logging events. * @return Stored callback message */ public static String getLoggingMessage(); // ===== Setting log configuration ===== public static void setLogConfigFile (String cfg_file); public static void setLogLevel (String level); // Available levels are: TRACE, DEBUG, INFO, WARN, ERROR, and OFF // ===== Setting file locations ===== public static void setRootFolder (String folder); // Root folder for config files, logs and project data. Default is ./ public static void setPurgeOnStart (boolean purge); // If set to true, the stored project list will be cleared the next time when the DLL is initialized // ===== Setting up JEA web service connection ===== public static boolean isUsingJEAOnline (); public static void setUseJEAOnline (boolean online); public static void setJEAOnlineUrl (String url); // Default url for debugging is "http://jess.ensims.com:2997/jea_web/api/"; public static void setJEAOnlineApiKey (String apikey); // JEA Web debugging API Key public static void setJEAOnlineUser (String email); // JEA Web debugging User (email) // ===== For JEA web connection, start/stop autochecker process ===== public static void startAutoChecker(String project_id); public static void stopAutoChecker();
Sequence diagram
A sequence diagram of DB - COM(vb6Events) - jea_ikvm_wrapper - JEA illustrating some of the key transactions between the components, including:
- Create/Control EA
- Raising simulation request
- Set simulation result
SendCommand
These are the set of commands for controlling EA process, including creating, starting, pausing, stopping and inquiring the EA engine.
Command Structure
A typical EA command is shown below. It contains four fields: command, project reference, problem definition, and EA configuration.
command
: the command string can be one of the following:Create
- Create an EA engineUpdate
- Update an existing EA engineStart
- Start an existing EA enginePause
- Pause operation of an existing EA engineResume
- Resume operationTerminate
- Terminate operation. The EA engine will stop after the current generationCancel
- Cancel the operation of the given EA engine. Engine will stop immediately without waiting for pending results.Reset
- Reset operation clears historical data of the project to allow optimisation starts at generation 0Delete
- Delete the given engine. This must be done before another engine with the same name can be createdStatus
- Get the status of the given engineReport
- Get a report from the given engineInfo
- Info operation to get version information of the JEA engine
projectID
: the original project ID forCreate
, or a reference to the existing project for othersproblem
: optimisation problem definition object used only inCreate
andUpdate
. In other commands it can be set to null or omitted.config
: algorithm configuration object used only inCreate
andUpdate
. In other commands it can be set to null or omitted.smData
: arbitrary data can be attached to the command. This is for storing information with the project. JEA does not make use of these data. AnsmData
object is required forCreate
andUpdate
commands. It can be empty, but not null.filter
: filters can be applied when querying the project archive. This feature is not currently available.
Example:
{ "command": "Pause", "projectID": "my_project", "problem": null, "config": null, "filter": null, "smData": null }
Null fields can be omitted in JSON objects.
Responses to SendCommand
The EA engine responds to a command with an Engine Response packet that contains an 'ok' field and a 'description' field, in the form as below:
ok
: Boolean field indicating if the command is accepted or successful executed.false
indicates errordescription
: Message set by the EA engine to provide textural information about how the command has been handled.
For example, an error status may look like this:
{ "ok": false, "description": "Engine does not exist" }
A normal (successful) status may look like this:
{ "ok": true, "description": "Engine for my_project has been created and configured." }
Transactions
Here are the details of the SendCommand transactions.
Create
Details of the problem definition and algorithm configuration objects are explained later. An example “Create” command is shown below:
{ "command": "Create", "projectID": "my_project", "problem": { "name": "Binh&Korn", "description": "Test constrained mo fucntion with two variables", "modelVersion": "1.0.0", "variables": [ { "name": "x", "caption": "x variable", "valueStr": "[0:0.01:1]", "maskStr": "[0.5:0.01:0.6]", "valueType": "Number" }, { "name": "y", "caption": "y variable", "valueStr": "[0:0.01:1]", "maskStr": null, "valueType": "Number" } ], "evalResults": [ { "name": "f1", "caption": "Objective function 1", "unit": "-" }, { "name": "f2", "caption": "Objective function 2", "unit": "-" }, { "name": "g1", "caption": "Constraint function 1", "unit": "-" }, { "name": "g2", "caption": "Constraint function 2", "unit": "-" } ], "userMetrics": [], "objectives": [ { "name": "o1", "caption": "Objective 1", "unit": "-", "direction": "Minimize", "formula": "f1" }, { "name": "o2", "caption": "Objective 2", "unit": "-", "direction": "Minimize", "formula": "f2" } ], "constraints": [ { "name": "c1", "caption": "Constraint 1", "unit": "-", "formula": "g1", "lb": 0, "ub": 25, "min": 0, "max": 1000, "weight": 1 }, { "name": "c2", "caption": "Contraint 2", "unit": "-", "formula": "g2", "lb": 7.7, "ub": 100000, "min": 0, "max": 1000000, "weight": 1 } ] }, "config": { "algorithm": "Sampling", "sampleSize": 0, "initSampleOption": "LHS", "initPopSize": 10, "evolvePopSize": 10, "maxPopSize": 10000, "mutationRate": 0.2, "crossoverRate": 1, "tournamentSize": 2, "objectiveBias": 0, "globalElitism": true, "elitismTolerance": 0, "maxGenerations": 100, "maxEvaluations": 1000, "maxComputingTime": 100, "maxWallTime": 24 } }
Update
The Update command is similar to the Create command except that it is used during the operation of the EA engine, to change problem definitions and/or the algorithm configuration on the fly.
The syntax of the traction is the same as Create
, except with the Update
command and the projectID referencing an existing project.
Start
Start the EA process. An example of this command is provided here:
{ "command": "Start", "projectID": "my_project" }
Pause
Pause the EA process. An example of this command is provided here:
{ "command": "Pause", "projectID": "my_project" }
Resume
Resume a paused the EA process. An example of this command is provided here:
{ "command": "Resume", "projectID": "my_project" }
Terminate
Terminate the EA process. An example of this command is provided here:
{ "command": "Terminate", "projectID": "my_project" }
Cancel
Cancel the EA process. Different to Terminate, Cancel stops the EA process immediately without waiting for pending simulation results. An example of this command is provided here:
{ "command": "Cancel", "projectID": "my_project" }
Reset
Reset the EA process. This operation clears all historical data and put the engine in its original “Created” state. An example of this command is provided here:
{ "command": "Reset", "projectID": "my_project" }
Delete
Delete the EA process and release the Project_ID. After deletion, all historical records will be lost.
Please note that JEA will action on the command immediately and does not provide an opportunity to confirm deletion. Confirmation must be done in the client's GUI.
An example of this command is provided here:
{ "command": "Delete", "projectID": "my_project" }
Status
Get the current status of the EA process. An example of this command is provided here:
{ "command": "Status", "projectID": "my_project" }
Unlike to other commands, the engine respond to Status
with its current status report. An Engine Status packet contains the following fields:
ok
- Status is ok or notprojectName
- reference to the current projectstatus
- Engine status code. It is one of the following, as shown in the Engine States Diagram:Unknown
- Engine does not exist or certain error statesCreated
- Engine created and configuredInitialzing
- Running the first generation, which is normally randomly sampled and can have a different population sizeEvolving
- Evolution on-goingPaused
- Evolution process on-holdTerminated
- Evolution process terminated, either on completion (including budget limit reached), or on user's commandCanceled
- Evolution process cancelled, on user's commandError
- Error state during creation
description
- Textual description of the statusGenerations
- number of generations has completed. This equals to the current/next generation number as JEA's zero-basedSolutions
- number of solutions have been explored.
An example status packet is as below:
{ "projectName": "my_project", "status": "Evolving", "generations": 1, "solutions": 10, "ok": true, "description": "Engine is running. Current generation: 0, solutions explored: 10" }
Report
Get the detailed report of the EA process. An example of this command is provided here:
{ "command": "Report", "projectID": "my_project" }
Return data of the report inquest contains the information of the current project settings, engine configuration, the latest population, and the statistics from the previous complete runs so far. If partial statistics, i.e. stats on the current set of solutions in the archive disregarding whether the run is complete or not, are required, add an empty filter
object in the Report command as below:
{ "command": "Report", "projectID": "my_project", "filter": {} }
Example report output is shown below.
{ "stats": { "descStats": { "x": { "max": 1, "mean": 0.5029411764705881, "stdDev": 0.3638891723702025, "min": 0, "variance": 0.13241532976827095, "n": 34, "sum": 17.100000000000005 }, ... }, "histograms": { "x": { "0.00714286": 14, "0.200000": 2, "0.300000": 3, "0.400000": 4, "0.500000": 1, "NaN": 0, "0.700000": 3, "0.800000": 5, "0.900000": 5, "1.00000": 3 }, ... }, "regressions": { "f1": { "adjRSquared": 1, "regParams": [ 0, 99.99999999999997, 1.6738389663652936e-15 ], "stdErrors": [ 4.869383557358312e-15, 7.473273002785428e-15, 8.015635396813993e-15 ], "pvalues": [ 1, 0, 0.8357327962958778 ], "residuals": [ 1.0658141036401503e-14, 7.105427357601002e-15, 7.105427357601002e-15, ... ], "stdCoeffs": [ -6.143958570009224e-17, 0.9999999999999997, 5.572302270711456e-17 ] }, ... }, "morris": {}, "sobol": {}, "invalidRemoved": 6 }, "problem": { ... }, "currentPopulation": { "batchJobID": "local", "newElites": [ "C-57_88", "C-52_16", ], "epoch": 0, "isRemote": false, "localResultFolder": "./", "elites": [ "C-57_88", "C-52_16", ], "solutions": [ "C-57_88", "C-52_16", "C-51_39", ... ] }, "lastGeneration": 0, "nsolutions": 10, "nelites": 8, "nnewElites": 8, "config": { ... }, "status": "Evolving", "projectName": "my_project", "description": "EA Engine report", "ok": true }
Info
Get the version information of the JEA engine. An example of this command is provided here:
{ "command": "Info", "projectID": null }
Example returned data are as below:
{ "Title": "JEA Web API", "Description": "an optimisation service provided by ENSIMS Ltd.", "Major": 1, "Minor": 0, "Revision": 0, "Release": "beta", "Update": 4, "Notice": "(C) 2016, Energy Simulation Solutions Ltd. All rights reserved." }
Events
SimulationRequest
An example of the simulation request command is shown below. A SimulationRequst packet has the eventType
field set to Request
. The projectName
field is used to identify the project of which the simulation cases belong. This is useful for multiple on-going optimisation runs. The name
field is for identify this particular submission (batch) within the project. It is important to keep track of the batch name, as it is used to cancel ongoing jobs.
jobSet
entries are a map between case IDs and variable value maps. The last field, eventType
, specifies that this is a SimulationRequest.
{ "eventType": "Request", "name": "Gen-0", "projectName": "my_project", "jobSet": { "C-26_27": { "x": 2.6, "y": 2.7 }, "C-3_5": { "x": 0.3, "y": 0.5 }, "C-16_17": { "x": 1.6, "y": 1.7 }, "C-31_0": { "x": 3.1, "y": 0 } } }
CancelRequest
The CancelRequest
events are invoked using the same callback delegate as the SimulationRequest
events. It is essentially an empty simulation request packet with eventType
set to CancelRequest
. This event is triggered when the EA engine is cancelled. The client can use this event to cancel the running simulation jobs. An example is shown here.
{ "eventType": "CancelRequest", "name": "Gen-0", "projectName": "my_project" }
StatusChanged
StatusChanged
events are fired when the engine's status has changed, e.g. when evolution progresses, paused, or terminated. Please note that in the current version, StatusChanged
events are only fired when the EA engine is live, i.e. a Start
command has been executed. At other times the status of the engine can be queried with a Status
command.
The event data packet contains eventType
, projectName
, previousStatus
and currentStatus
, as shown in the example below.
{ "eventType": "StatusChanged", "projectName": "my_project", "previous": { "projectName": "my_project", "status": "Initialzing", "generations": 0, "solutions": 0, "ok": true, "description": "Initializing the first set of solutions." }, "current": { "projectName": "my_project", "status": "Evolving", "generations": 1, "solutions": 10, "ok": true, "description": "Engine is running. Current generation: 0, solutions explored: 10" } }
Logging
If a logging callback delegate is set, Logging
events will be fired when the engine writes log entries. A logging event data packet contains the log text only. It may contain multiple lines in one entries and may contain multiple entries.
Logs are not associated with particular projects. They are for development and diagnostics only. They should not be used for reporting progress to users.
Set Callback delegate
Three functions are provided for client to specify callback delegates for the events. These functions are:
void setCallback(RunnableDelegate Callback)
- Set delegate forRequest
andCancelRequest
eventsvoid setStatusChangedCallback(RunnableDelegate Callback)
- Set delegate forStatusChanged
eventsvoid setLoggingCallback(RunnableDelegate Callback)
- Set delegate forLogging
events
If delegates are not set for StatusChanged
and Logging
events, the events will not be fired by JEA. Since it is not possible to identify the event type by calling the delegate, it is necessary to have separate delegates for each events, so that they can retrieve the correct event messages.
Retrieve Callback message
Since callback delegate does not take arguments, once a callback is received, it has to make an API call to retrieve the event message. There are three API functions provided for retrieving different event's messages:
String getCallbackMessage(String prj_id)
- retrieveRequest
andCancelRequest
messages. Requires the project name to be specified.String getStatusChangedMessage(String prj_id)
- retrieveStatusChanged
messages. Requires the project name to be specified.String getLoggingMessage()
- retrieveLogging
messages.
The contents of the event messages are explained in events details.
SetEvaluationResult
Once simulation request event is received and the simulation jobs have been executed, the client calls the API function String setEvaluationResult (String jsoncmd)
to send the results back to JEA. Format of the data packet of the simulation results are explained here. The return string of this function is an Engine Response object in JSON.
SimulationReturn
The JSON encoding of a SimulationReturn
object would look like below. It is identified by its name
and projectName
, corresponding to those set in the SimulationRequest
object. The resultSet
is a map between case IDs and the results maps.
{ "name": "gen0", "projectName": "my_project", "resultSet": { "C-26_27": { "f1": 56.2, "f2": 11.049999999999999, "g1": 13.05, "g2": 61.650000000000006 }, "C-3_5": { "f1": 1.3599999999999999, "f2": 42.34, "g1": 22.340000000000003, "g2": 71.54 }, ... }, "cpuTime": "0" }
Simulation Failures
In case of simulation failure, return empty objects for the affected cases, such as the case C-26_27
in the next example.
{ "name": "gen0", "projectName": "my_project", "resultSet": { "C-26_27": { }, "C-3_5": { "f1": 1.3599999999999999, "f2": 42.34, "g1": 22.340000000000003, "g2": 71.54 }, ... }, "cpuTime": "0" }