Training a Neural ODE to Model Gravitational Waveforms
This code is adapted from Astroinformatics/ScientificMachineLearning
The code has been minimally adapted from Keith et. al. 2021 which originally used Flux.jl
Package Imports
julia
using Lux, ComponentArrays, LineSearches, OrdinaryDiffEqLowOrderRK, Optimization,
OptimizationOptimJL, Printf, Random, SciMLSensitivity
using CairoMakiePrecompiling Lux...
549.1 ms ✓ ConcreteStructs
474.4 ms ✓ SIMDTypes
489.5 ms ✓ Reexport
512.3 ms ✓ Future
503.9 ms ✓ IfElse
550.0 ms ✓ CEnum
554.3 ms ✓ ArgCheck
570.5 ms ✓ OpenLibm_jll
570.7 ms ✓ ManualMemory
705.7 ms ✓ CompilerSupportLibraries_jll
725.6 ms ✓ Requires
780.7 ms ✓ Statistics
842.1 ms ✓ EnzymeCore
883.2 ms ✓ ADTypes
477.1 ms ✓ CommonWorldInvalidations
476.3 ms ✓ FastClosures
607.9 ms ✓ ConstructionBase
535.1 ms ✓ StaticArraysCore
695.4 ms ✓ JLLWrappers
599.9 ms ✓ NaNMath
800.6 ms ✓ Compat
591.6 ms ✓ Adapt
1368.8 ms ✓ IrrationalConstants
552.9 ms ✓ ADTypes → ADTypesEnzymeCoreExt
917.4 ms ✓ CpuId
937.3 ms ✓ DocStringExtensions
527.5 ms ✓ ConstructionBase → ConstructionBaseLinearAlgebraExt
540.2 ms ✓ ADTypes → ADTypesConstructionBaseExt
557.5 ms ✓ DiffResults
518.5 ms ✓ Compat → CompatLinearAlgebraExt
1185.8 ms ✓ ThreadingUtilities
518.4 ms ✓ EnzymeCore → AdaptExt
685.0 ms ✓ Hwloc_jll
611.0 ms ✓ GPUArraysCore
635.7 ms ✓ ArrayInterface
1103.0 ms ✓ Static
776.1 ms ✓ OpenSpecFun_jll
715.3 ms ✓ LogExpFunctions
2495.9 ms ✓ UnsafeAtomics
419.7 ms ✓ ArrayInterface → ArrayInterfaceStaticArraysCoreExt
463.1 ms ✓ ArrayInterface → ArrayInterfaceGPUArraysCoreExt
527.5 ms ✓ BitTwiddlingConvenienceFunctions
823.2 ms ✓ Functors
498.7 ms ✓ Atomix
1114.5 ms ✓ CPUSummary
2758.1 ms ✓ MacroTools
1463.9 ms ✓ ChainRulesCore
1137.9 ms ✓ MLDataDevices
513.9 ms ✓ ArrayInterface → ArrayInterfaceChainRulesCoreExt
539.7 ms ✓ ADTypes → ADTypesChainRulesCoreExt
1886.6 ms ✓ StaticArrayInterface
841.2 ms ✓ PolyesterWeave
857.8 ms ✓ CommonSubexpressions
551.1 ms ✓ CloseOpenIntervals
2564.7 ms ✓ Hwloc
730.5 ms ✓ MLDataDevices → MLDataDevicesChainRulesCoreExt
609.4 ms ✓ LayoutPointers
1472.0 ms ✓ Optimisers
1573.7 ms ✓ LogExpFunctions → LogExpFunctionsChainRulesCoreExt
1708.7 ms ✓ Setfield
1918.9 ms ✓ DispatchDoctor
443.8 ms ✓ Optimisers → OptimisersAdaptExt
447.8 ms ✓ Optimisers → OptimisersEnzymeCoreExt
3121.8 ms ✓ SpecialFunctions
1009.4 ms ✓ StrideArraysCore
452.1 ms ✓ DispatchDoctor → DispatchDoctorEnzymeCoreExt
668.2 ms ✓ DispatchDoctor → DispatchDoctorChainRulesCoreExt
613.3 ms ✓ DiffRules
764.3 ms ✓ Polyester
1246.2 ms ✓ LuxCore
490.1 ms ✓ LuxCore → LuxCoreMLDataDevicesExt
492.0 ms ✓ LuxCore → LuxCoreSetfieldExt
494.6 ms ✓ LuxCore → LuxCoreFunctorsExt
575.8 ms ✓ LuxCore → LuxCoreEnzymeCoreExt
638.2 ms ✓ LuxCore → LuxCoreChainRulesCoreExt
1778.3 ms ✓ SpecialFunctions → SpecialFunctionsChainRulesCoreExt
2778.5 ms ✓ WeightInitializers
7409.8 ms ✓ StaticArrays
611.7 ms ✓ ConstructionBase → ConstructionBaseStaticArraysExt
610.7 ms ✓ Adapt → AdaptStaticArraysExt
950.1 ms ✓ WeightInitializers → WeightInitializersChainRulesCoreExt
632.2 ms ✓ StaticArrays → StaticArraysChainRulesCoreExt
655.2 ms ✓ StaticArrays → StaticArraysStatisticsExt
699.9 ms ✓ StaticArrayInterface → StaticArrayInterfaceStaticArraysExt
3598.1 ms ✓ ForwardDiff
844.0 ms ✓ ForwardDiff → ForwardDiffStaticArraysExt
3168.8 ms ✓ KernelAbstractions
624.1 ms ✓ KernelAbstractions → LinearAlgebraExt
687.4 ms ✓ KernelAbstractions → EnzymeExt
5052.3 ms ✓ NNlib
805.4 ms ✓ NNlib → NNlibEnzymeCoreExt
901.0 ms ✓ NNlib → NNlibForwardDiffExt
5399.6 ms ✓ LuxLib
8890.0 ms ✓ Lux
94 dependencies successfully precompiled in 33 seconds. 26 already precompiled.
Precompiling ComponentArrays...
859.8 ms ✓ ComponentArrays
1 dependency successfully precompiled in 1 seconds. 57 already precompiled.
Precompiling MLDataDevicesComponentArraysExt...
479.7 ms ✓ MLDataDevices → MLDataDevicesComponentArraysExt
1 dependency successfully precompiled in 1 seconds. 60 already precompiled.
Precompiling LuxComponentArraysExt...
502.8 ms ✓ ComponentArrays → ComponentArraysOptimisersExt
1455.8 ms ✓ Lux → LuxComponentArraysExt
1861.4 ms ✓ ComponentArrays → ComponentArraysKernelAbstractionsExt
3 dependencies successfully precompiled in 2 seconds. 122 already precompiled.
Precompiling LineSearches...
285.7 ms ✓ UnPack
466.5 ms ✓ OrderedCollections
531.0 ms ✓ Serialization
576.5 ms ✓ FiniteDiff
387.4 ms ✓ Parameters
1603.4 ms ✓ Distributed
991.2 ms ✓ NLSolversBase
1713.9 ms ✓ LineSearches
8 dependencies successfully precompiled in 5 seconds. 47 already precompiled.
Precompiling FiniteDiffStaticArraysExt...
555.9 ms ✓ FiniteDiff → FiniteDiffStaticArraysExt
1 dependency successfully precompiled in 1 seconds. 23 already precompiled.
Precompiling OrdinaryDiffEqLowOrderRK...
460.7 ms ✓ FastPower
474.6 ms ✓ IteratorInterfaceExtensions
471.2 ms ✓ DataValueInterfaces
477.6 ms ✓ CommonSolve
504.0 ms ✓ SimpleUnPack
503.8 ms ✓ MuladdMacro
518.2 ms ✓ CompositionsBase
533.4 ms ✓ EnumX
543.9 ms ✓ DataAPI
555.5 ms ✓ ExprTools
602.6 ms ✓ SciMLStructures
633.4 ms ✓ InverseFunctions
803.8 ms ✓ TruncatedStacktraces
381.2 ms ✓ TableTraits
985.0 ms ✓ FunctionWrappers
547.2 ms ✓ RuntimeGeneratedFunctions
479.7 ms ✓ CompositionsBase → CompositionsBaseInverseFunctionsExt
514.6 ms ✓ LogExpFunctions → LogExpFunctionsInverseFunctionsExt
536.3 ms ✓ InverseFunctions → InverseFunctionsDatesExt
1281.0 ms ✓ FillArrays
396.3 ms ✓ FunctionWrappersWrappers
958.4 ms ✓ FastPower → FastPowerForwardDiffExt
998.1 ms ✓ PreallocationTools
1001.2 ms ✓ FastBroadcast
810.5 ms ✓ Tables
425.2 ms ✓ FillArrays → FillArraysStatisticsExt
1759.1 ms ✓ RecipesBase
1759.3 ms ✓ DataStructures
2573.8 ms ✓ Accessors
748.1 ms ✓ Accessors → AccessorsDatesExt
1294.1 ms ✓ SymbolicIndexingInterface
1627.5 ms ✓ SciMLOperators
505.1 ms ✓ SciMLOperators → SciMLOperatorsStaticArraysCoreExt
1961.2 ms ✓ RecursiveArrayTools
749.6 ms ✓ RecursiveArrayTools → RecursiveArrayToolsForwardDiffExt
794.2 ms ✓ RecursiveArrayTools → RecursiveArrayToolsFastBroadcastExt
12178.2 ms ✓ MLStyle
7285.9 ms ✓ Expronicon
10923.0 ms ✓ SciMLBase
5789.1 ms ✓ DiffEqBase
4387.1 ms ✓ OrdinaryDiffEqCore
1467.5 ms ✓ OrdinaryDiffEqCore → OrdinaryDiffEqCoreEnzymeCoreExt
4072.4 ms ✓ OrdinaryDiffEqLowOrderRK
43 dependencies successfully precompiled in 46 seconds. 82 already precompiled.
Precompiling AccessorsStaticArraysExt...
648.0 ms ✓ Accessors → AccessorsStaticArraysExt
1 dependency successfully precompiled in 1 seconds. 20 already precompiled.
Precompiling MLDataDevicesRecursiveArrayToolsExt...
592.2 ms ✓ MLDataDevices → MLDataDevicesRecursiveArrayToolsExt
1 dependency successfully precompiled in 1 seconds. 48 already precompiled.
Precompiling ComponentArraysRecursiveArrayToolsExt...
693.0 ms ✓ ComponentArrays → ComponentArraysRecursiveArrayToolsExt
1 dependency successfully precompiled in 1 seconds. 81 already precompiled.
Precompiling ComponentArraysSciMLBaseExt...
947.1 ms ✓ SciMLBase → SciMLBaseChainRulesCoreExt
1084.2 ms ✓ ComponentArrays → ComponentArraysSciMLBaseExt
2 dependencies successfully precompiled in 1 seconds. 97 already precompiled.
Precompiling DiffEqBaseChainRulesCoreExt...
1505.7 ms ✓ DiffEqBase → DiffEqBaseChainRulesCoreExt
1 dependency successfully precompiled in 2 seconds. 125 already precompiled.
Precompiling MLDataDevicesFillArraysExt...
425.6 ms ✓ MLDataDevices → MLDataDevicesFillArraysExt
1 dependency successfully precompiled in 1 seconds. 15 already precompiled.
Precompiling Optimization...
429.7 ms ✓ SuiteSparse_jll
456.3 ms ✓ ProgressLogging
504.4 ms ✓ AbstractTrees
534.5 ms ✓ LoggingExtras
635.8 ms ✓ L_BFGS_B_jll
853.4 ms ✓ DifferentiationInterface
863.6 ms ✓ ProgressMeter
381.5 ms ✓ LeftChildRightSiblingTrees
496.1 ms ✓ LBFGSB
446.5 ms ✓ ConsoleProgressMonitor
632.3 ms ✓ TerminalLoggers
3644.1 ms ✓ SparseArrays
603.3 ms ✓ SuiteSparse
627.3 ms ✓ ArrayInterface → ArrayInterfaceSparseArraysExt
635.7 ms ✓ Statistics → SparseArraysExt
661.0 ms ✓ DifferentiationInterface → DifferentiationInterfaceSparseArraysExt
757.1 ms ✓ FillArrays → FillArraysSparseArraysExt
862.4 ms ✓ SciMLOperators → SciMLOperatorsSparseArraysExt
923.8 ms ✓ RecursiveArrayTools → RecursiveArrayToolsSparseArraysExt
1208.2 ms ✓ SparseMatrixColorings
854.3 ms ✓ PDMats
854.6 ms ✓ DifferentiationInterface → DifferentiationInterfaceSparseMatrixColoringsExt
627.0 ms ✓ FillArrays → FillArraysPDMatsExt
3470.3 ms ✓ SparseConnectivityTracer
2110.6 ms ✓ OptimizationBase
1938.5 ms ✓ Optimization
26 dependencies successfully precompiled in 12 seconds. 78 already precompiled.
Precompiling ChainRulesCoreSparseArraysExt...
616.0 ms ✓ ChainRulesCore → ChainRulesCoreSparseArraysExt
1 dependency successfully precompiled in 1 seconds. 11 already precompiled.
Precompiling SparseArraysExt...
889.3 ms ✓ KernelAbstractions → SparseArraysExt
1 dependency successfully precompiled in 1 seconds. 26 already precompiled.
Precompiling MLDataDevicesSparseArraysExt...
649.7 ms ✓ MLDataDevices → MLDataDevicesSparseArraysExt
1 dependency successfully precompiled in 1 seconds. 17 already precompiled.
Precompiling FiniteDiffSparseArraysExt...
578.7 ms ✓ FiniteDiff → FiniteDiffSparseArraysExt
1 dependency successfully precompiled in 1 seconds. 18 already precompiled.
Precompiling DiffEqBaseSparseArraysExt...
1610.7 ms ✓ DiffEqBase → DiffEqBaseSparseArraysExt
1 dependency successfully precompiled in 2 seconds. 125 already precompiled.
Precompiling DifferentiationInterfaceFiniteDiffExt...
392.8 ms ✓ DifferentiationInterface → DifferentiationInterfaceFiniteDiffExt
1 dependency successfully precompiled in 0 seconds. 17 already precompiled.
Precompiling DifferentiationInterfaceChainRulesCoreExt...
400.1 ms ✓ DifferentiationInterface → DifferentiationInterfaceChainRulesCoreExt
1 dependency successfully precompiled in 1 seconds. 11 already precompiled.
Precompiling DifferentiationInterfaceStaticArraysExt...
566.6 ms ✓ DifferentiationInterface → DifferentiationInterfaceStaticArraysExt
1 dependency successfully precompiled in 1 seconds. 10 already precompiled.
Precompiling DifferentiationInterfaceForwardDiffExt...
752.9 ms ✓ DifferentiationInterface → DifferentiationInterfaceForwardDiffExt
1 dependency successfully precompiled in 1 seconds. 41 already precompiled.
Precompiling SparseConnectivityTracerSpecialFunctionsExt...
1140.2 ms ✓ SparseConnectivityTracer → SparseConnectivityTracerLogExpFunctionsExt
1534.9 ms ✓ SparseConnectivityTracer → SparseConnectivityTracerSpecialFunctionsExt
2 dependencies successfully precompiled in 2 seconds. 39 already precompiled.
Precompiling SparseConnectivityTracerNNlibExt...
1607.8 ms ✓ SparseConnectivityTracer → SparseConnectivityTracerNNlibExt
1 dependency successfully precompiled in 2 seconds. 46 already precompiled.
Precompiling SparseConnectivityTracerNaNMathExt...
1207.3 ms ✓ SparseConnectivityTracer → SparseConnectivityTracerNaNMathExt
1 dependency successfully precompiled in 1 seconds. 18 already precompiled.
Precompiling OptimizationFiniteDiffExt...
349.2 ms ✓ OptimizationBase → OptimizationFiniteDiffExt
1 dependency successfully precompiled in 1 seconds. 97 already precompiled.
Precompiling OptimizationForwardDiffExt...
626.1 ms ✓ OptimizationBase → OptimizationForwardDiffExt
1 dependency successfully precompiled in 1 seconds. 110 already precompiled.
Precompiling OptimizationMLDataDevicesExt...
1373.9 ms ✓ OptimizationBase → OptimizationMLDataDevicesExt
1 dependency successfully precompiled in 2 seconds. 97 already precompiled.
Precompiling HwlocTrees...
497.6 ms ✓ Hwloc → HwlocTrees
1 dependency successfully precompiled in 1 seconds. 10 already precompiled.
Precompiling OptimizationOptimJL...
314.8 ms ✓ PtrArrays
374.7 ms ✓ StatsAPI
447.4 ms ✓ PositiveFactorizations
438.6 ms ✓ Missings
517.9 ms ✓ SortingAlgorithms
425.9 ms ✓ AliasTables
2152.4 ms ✓ StatsBase
3028.3 ms ✓ Optim
12222.3 ms ✓ OptimizationOptimJL
9 dependencies successfully precompiled in 18 seconds. 131 already precompiled.
Precompiling SciMLSensitivity...
510.3 ms ✓ RealDot
523.4 ms ✓ StructIO
589.2 ms ✓ PoissonRandom
588.7 ms ✓ Scratch
777.1 ms ✓ AbstractFFTs
911.3 ms ✓ SparseInverseSubset
1134.2 ms ✓ StructArrays
1304.0 ms ✓ Cassette
808.4 ms ✓ Rmath_jll
1346.8 ms ✓ RandomNumbers
1417.9 ms ✓ KLU
1473.9 ms ✓ OffsetArrays
924.3 ms ✓ oneTBB_jll
862.9 ms ✓ ResettableStacks
1781.6 ms ✓ FastLapackInterface
1415.0 ms ✓ ZygoteRules
1590.8 ms ✓ LazyArtifacts
631.6 ms ✓ AbstractFFTs → AbstractFFTsChainRulesCoreExt
1144.2 ms ✓ HostCPUFeatures
603.6 ms ✓ StructArrays → StructArraysAdaptExt
1701.9 ms ✓ QuadGK
1828.0 ms ✓ HypergeometricFunctions
659.1 ms ✓ StructArrays → StructArraysGPUArraysCoreExt
3246.5 ms ✓ TimerOutputs
743.5 ms ✓ Accessors → AccessorsStructArraysExt
1005.8 ms ✓ StructArrays → StructArraysSparseArraysExt
1150.9 ms ✓ StructArrays → StructArraysStaticArraysExt
3063.6 ms ✓ IRTools
738.4 ms ✓ FunctionProperties
1003.9 ms ✓ RecursiveArrayTools → RecursiveArrayToolsStructArraysExt
547.3 ms ✓ OffsetArrays → OffsetArraysAdaptExt
683.2 ms ✓ StaticArrayInterface → StaticArrayInterfaceOffsetArraysExt
1170.1 ms ✓ Rmath
1152.7 ms ✓ Random123
3025.1 ms ✓ ObjectFile
5290.1 ms ✓ Test
1767.6 ms ✓ IntelOpenMP_jll
2186.7 ms ✓ Enzyme_jll
4423.2 ms ✓ SciMLJacobianOperators
2091.2 ms ✓ LLVMExtra_jll
758.4 ms ✓ InverseFunctions → InverseFunctionsTestExt
845.7 ms ✓ Accessors → AccessorsTestExt
2409.2 ms ✓ StatsFuns
1550.4 ms ✓ MKL_jll
1603.0 ms ✓ Sparspak
1794.6 ms ✓ AbstractFFTs → AbstractFFTsTestExt
915.9 ms ✓ StatsFuns → StatsFunsInverseFunctionsExt
8161.0 ms ✓ Krylov
6665.2 ms ✓ DiffEqCallbacks
6931.0 ms ✓ Tracker
2085.2 ms ✓ StatsFuns → StatsFunsChainRulesCoreExt
1598.0 ms ✓ ArrayInterface → ArrayInterfaceTrackerExt
1600.8 ms ✓ FastPower → FastPowerTrackerExt
1609.6 ms ✓ DifferentiationInterface → DifferentiationInterfaceTrackerExt
1672.4 ms ✓ RecursiveArrayTools → RecursiveArrayToolsTrackerExt
1848.8 ms ✓ Tracker → TrackerPDMatsExt
2542.6 ms ✓ DiffEqBase → DiffEqBaseTrackerExt
7049.6 ms ✓ ChainRules
779.9 ms ✓ ArrayInterface → ArrayInterfaceChainRulesExt
8395.8 ms ✓ VectorizationBase
5632.8 ms ✓ Distributions
7165.6 ms ✓ LLVM
1157.6 ms ✓ SLEEFPirates
1506.4 ms ✓ Distributions → DistributionsTestExt
1530.8 ms ✓ Distributions → DistributionsChainRulesCoreExt
2067.5 ms ✓ DiffEqBase → DiffEqBaseDistributionsExt
14582.4 ms ✓ ArrayLayouts
1937.7 ms ✓ UnsafeAtomics → UnsafeAtomicsLLVM
779.4 ms ✓ ArrayLayouts → ArrayLayoutsSparseArraysExt
2369.6 ms ✓ GPUArrays
2424.6 ms ✓ LazyArrays
3886.3 ms ✓ DiffEqNoiseProcess
1335.4 ms ✓ LazyArrays → LazyArraysStaticArraysExt
17987.7 ms ✓ ReverseDiff
3493.5 ms ✓ FastPower → FastPowerReverseDiffExt
3646.4 ms ✓ DifferentiationInterface → DifferentiationInterfaceReverseDiffExt
4725.7 ms ✓ ArrayInterface → ArrayInterfaceReverseDiffExt
4743.0 ms ✓ PreallocationTools → PreallocationToolsReverseDiffExt
4940.4 ms ✓ DiffEqBase → DiffEqBaseReverseDiffExt
5027.8 ms ✓ DiffEqNoiseProcess → DiffEqNoiseProcessReverseDiffExt
18375.8 ms ✓ GPUCompiler
21045.5 ms ✓ LoopVectorization
1172.9 ms ✓ LoopVectorization → SpecialFunctionsExt
1300.4 ms ✓ LoopVectorization → ForwardDiffExt
3954.7 ms ✓ TriangularSolve
28204.1 ms ✓ Zygote
1596.6 ms ✓ DifferentiationInterface → DifferentiationInterfaceZygoteExt
1915.6 ms ✓ Zygote → ZygoteTrackerExt
3109.4 ms ✓ RecursiveArrayTools → RecursiveArrayToolsZygoteExt
3502.1 ms ✓ SciMLBase → SciMLBaseZygoteExt
5337.4 ms ✓ RecursiveArrayTools → RecursiveArrayToolsReverseDiffExt
16013.1 ms ✓ RecursiveFactorization
29109.1 ms ✓ LinearSolve
2559.6 ms ✓ LinearSolve → LinearSolveRecursiveArrayToolsExt
2615.4 ms ✓ LinearSolve → LinearSolveEnzymeExt
4150.2 ms ✓ LinearSolve → LinearSolveKernelAbstractionsExt
198408.8 ms ✓ Enzyme
6967.2 ms ✓ Enzyme → EnzymeGPUArraysCoreExt
6996.1 ms ✓ FastPower → FastPowerEnzymeExt
7025.9 ms ✓ QuadGK → QuadGKEnzymeExt
7079.3 ms ✓ DifferentiationInterface → DifferentiationInterfaceEnzymeExt
7163.4 ms ✓ Enzyme → EnzymeLogExpFunctionsExt
7244.9 ms ✓ Enzyme → EnzymeSpecialFunctionsExt
17482.0 ms ✓ Enzyme → EnzymeStaticArraysExt
19109.4 ms ✓ Enzyme → EnzymeChainRulesCoreExt
17366.1 ms ✓ DiffEqBase → DiffEqBaseEnzymeExt
29139.0 ms ✓ SciMLSensitivity
107 dependencies successfully precompiled in 278 seconds. 181 already precompiled.
Precompiling LuxLibSLEEFPiratesExt...
2326.4 ms ✓ LuxLib → LuxLibSLEEFPiratesExt
1 dependency successfully precompiled in 3 seconds. 108 already precompiled.
Precompiling LuxLibLoopVectorizationExt...
4594.8 ms ✓ LuxLib → LuxLibLoopVectorizationExt
1 dependency successfully precompiled in 5 seconds. 116 already precompiled.
Precompiling LuxLibEnzymeExt...
1297.6 ms ✓ LuxLib → LuxLibEnzymeExt
1 dependency successfully precompiled in 2 seconds. 130 already precompiled.
Precompiling LuxEnzymeExt...
7408.7 ms ✓ Lux → LuxEnzymeExt
1 dependency successfully precompiled in 8 seconds. 146 already precompiled.
Precompiling OptimizationEnzymeExt...
20194.5 ms ✓ OptimizationBase → OptimizationEnzymeExt
1 dependency successfully precompiled in 20 seconds. 109 already precompiled.
Precompiling MLDataDevicesTrackerExt...
1144.6 ms ✓ MLDataDevices → MLDataDevicesTrackerExt
1 dependency successfully precompiled in 1 seconds. 70 already precompiled.
Precompiling LuxLibTrackerExt...
1075.5 ms ✓ LuxCore → LuxCoreArrayInterfaceTrackerExt
3144.4 ms ✓ LuxLib → LuxLibTrackerExt
2 dependencies successfully precompiled in 3 seconds. 111 already precompiled.
Precompiling LuxTrackerExt...
2023.2 ms ✓ Lux → LuxTrackerExt
1 dependency successfully precompiled in 2 seconds. 125 already precompiled.
Precompiling ComponentArraysTrackerExt...
1135.0 ms ✓ ComponentArrays → ComponentArraysTrackerExt
1 dependency successfully precompiled in 1 seconds. 81 already precompiled.
Precompiling MLDataDevicesReverseDiffExt...
3495.7 ms ✓ MLDataDevices → MLDataDevicesReverseDiffExt
1 dependency successfully precompiled in 4 seconds. 61 already precompiled.
Precompiling LuxLibReverseDiffExt...
3384.9 ms ✓ LuxCore → LuxCoreArrayInterfaceReverseDiffExt
4296.8 ms ✓ LuxLib → LuxLibReverseDiffExt
2 dependencies successfully precompiled in 5 seconds. 109 already precompiled.
Precompiling ComponentArraysReverseDiffExt...
3560.9 ms ✓ ComponentArrays → ComponentArraysReverseDiffExt
1 dependency successfully precompiled in 4 seconds. 69 already precompiled.
Precompiling OptimizationReverseDiffExt...
3387.3 ms ✓ OptimizationBase → OptimizationReverseDiffExt
1 dependency successfully precompiled in 4 seconds. 130 already precompiled.
Precompiling LuxReverseDiffExt...
4377.7 ms ✓ Lux → LuxReverseDiffExt
1 dependency successfully precompiled in 5 seconds. 126 already precompiled.
Precompiling MLDataDevicesChainRulesExt...
793.6 ms ✓ MLDataDevices → MLDataDevicesChainRulesExt
1 dependency successfully precompiled in 1 seconds. 40 already precompiled.
Precompiling MLDataDevicesZygoteExt...
1289.0 ms ✓ MLDataDevices → MLDataDevicesGPUArraysExt
1551.6 ms ✓ MLDataDevices → MLDataDevicesZygoteExt
2 dependencies successfully precompiled in 2 seconds. 92 already precompiled.
Precompiling LuxZygoteExt...
1347.7 ms ✓ WeightInitializers → WeightInitializersGPUArraysExt
2687.7 ms ✓ Lux → LuxZygoteExt
2 dependencies successfully precompiled in 3 seconds. 162 already precompiled.
Precompiling ComponentArraysZygoteExt...
1524.7 ms ✓ ComponentArrays → ComponentArraysGPUArraysExt
1538.3 ms ✓ ComponentArrays → ComponentArraysZygoteExt
2 dependencies successfully precompiled in 2 seconds. 98 already precompiled.
Precompiling OptimizationZygoteExt...
2155.5 ms ✓ OptimizationBase → OptimizationZygoteExt
1 dependency successfully precompiled in 2 seconds. 142 already precompiled.
Precompiling CairoMakie...
505.8 ms ✓ IndirectArrays
487.4 ms ✓ PolygonOps
557.2 ms ✓ GeoFormatTypes
554.6 ms ✓ PCRE2_jll
584.4 ms ✓ Contour
610.3 ms ✓ TriplotBase
616.7 ms ✓ TensorCore
615.8 ms ✓ StableRNGs
643.4 ms ✓ Extents
660.1 ms ✓ PaddedViews
689.1 ms ✓ RoundingEmulator
698.3 ms ✓ Observables
830.3 ms ✓ TranscodingStreams
437.1 ms ✓ CRC32c
544.1 ms ✓ IntervalSets → IntervalSetsStatisticsExt
540.2 ms ✓ LazyModules
1140.5 ms ✓ Grisu
593.0 ms ✓ StackViews
531.5 ms ✓ ConstructionBase → ConstructionBaseIntervalSetsExt
646.1 ms ✓ Inflate
605.4 ms ✓ RelocatableFolders
508.8 ms ✓ SignedDistanceFields
1544.0 ms ✓ Format
956.6 ms ✓ SharedArrays
964.4 ms ✓ WoodburyMatrices
1053.5 ms ✓ AxisArrays
861.5 ms ✓ OpenSSL_jll
812.2 ms ✓ Graphite2_jll
808.4 ms ✓ LLVMOpenMP_jll
836.2 ms ✓ Libmount_jll
839.2 ms ✓ Bzip2_jll
814.4 ms ✓ Xorg_libXau_jll
2333.2 ms ✓ AdaptivePredicates
856.4 ms ✓ libpng_jll
1539.3 ms ✓ SimpleTraits
868.0 ms ✓ libfdk_aac_jll
827.7 ms ✓ Giflib_jll
842.9 ms ✓ Imath_jll
856.3 ms ✓ LAME_jll
850.8 ms ✓ LERC_jll
881.6 ms ✓ EarCut_jll
837.8 ms ✓ CRlibm_jll
916.6 ms ✓ JpegTurbo_jll
818.7 ms ✓ Ogg_jll
924.1 ms ✓ XZ_jll
825.3 ms ✓ Xorg_libXdmcp_jll
859.8 ms ✓ x265_jll
894.2 ms ✓ x264_jll
883.3 ms ✓ libaom_jll
880.8 ms ✓ Zstd_jll
888.6 ms ✓ Expat_jll
2343.6 ms ✓ UnicodeFun
868.7 ms ✓ LZO_jll
823.0 ms ✓ Xorg_xtrans_jll
945.4 ms ✓ Opus_jll
872.1 ms ✓ Libgpg_error_jll
990.0 ms ✓ Libiconv_jll
889.2 ms ✓ isoband_jll
970.3 ms ✓ Libffi_jll
740.5 ms ✓ Xorg_libpthread_stubs_jll
3392.4 ms ✓ FixedPointNumbers
906.7 ms ✓ FFTW_jll
820.3 ms ✓ Libuuid_jll
950.6 ms ✓ FriBidi_jll
612.7 ms ✓ Showoff
642.2 ms ✓ MosaicViews
893.8 ms ✓ Pixman_jll
909.6 ms ✓ FreeType2_jll
1012.6 ms ✓ AxisAlgorithms
1540.2 ms ✓ FilePathsBase
1012.4 ms ✓ OpenEXR_jll
911.9 ms ✓ libsixel_jll
933.1 ms ✓ libvorbis_jll
912.6 ms ✓ Libtiff_jll
859.3 ms ✓ Libgcrypt_jll
1543.8 ms ✓ GeoInterface
626.5 ms ✓ Isoband
946.9 ms ✓ XML2_jll
562.4 ms ✓ Ratios → RatiosFixedPointNumbersExt
726.3 ms ✓ FilePathsBase → FilePathsBaseMmapExt
1104.0 ms ✓ Fontconfig_jll
868.8 ms ✓ Gettext_jll
1096.7 ms ✓ FilePaths
935.4 ms ✓ XSLT_jll
1387.7 ms ✓ FreeType
1701.9 ms ✓ FilePathsBase → FilePathsBaseTestExt
830.0 ms ✓ Glib_jll
3304.0 ms ✓ IntervalArithmetic
3041.1 ms ✓ ColorTypes
2907.9 ms ✓ Interpolations
1685.0 ms ✓ Xorg_libxcb_jll
759.6 ms ✓ IntervalArithmetic → IntervalArithmeticIntervalSetsExt
4800.2 ms ✓ PkgVersion
819.2 ms ✓ Xorg_libX11_jll
5433.3 ms ✓ FileIO
642.6 ms ✓ Xorg_libXrender_jll
870.8 ms ✓ Xorg_libXext_jll
2253.9 ms ✓ ColorVectorSpace
1063.5 ms ✓ Libglvnd_jll
1942.6 ms ✓ QOI
1116.9 ms ✓ Cairo_jll
855.2 ms ✓ ColorVectorSpace → SpecialFunctionsExt
1379.1 ms ✓ HarfBuzz_jll
1574.3 ms ✓ libwebp_jll
7477.8 ms ✓ GeometryBasics
5783.0 ms ✓ Colors
10467.1 ms ✓ SIMD
1587.7 ms ✓ libass_jll
8999.6 ms ✓ FFTW
6560.5 ms ✓ ExactPredicates
1766.3 ms ✓ Pango_jll
785.3 ms ✓ Graphics
811.6 ms ✓ Animations
1872.6 ms ✓ Packing
2126.5 ms ✓ ShaderAbstractions
1339.3 ms ✓ FFMPEG_jll
1764.8 ms ✓ ColorBrewer
2293.3 ms ✓ OpenEXR
1539.9 ms ✓ Cairo
2735.0 ms ✓ FreeTypeAbstraction
2587.8 ms ✓ KernelDensity
5076.7 ms ✓ MakieCore
4316.1 ms ✓ ColorSchemes
7079.6 ms ✓ GridLayoutBase
6127.6 ms ✓ DelaunayTriangulation
22924.3 ms ✓ Unitful
954.0 ms ✓ Unitful → InverseFunctionsUnitfulExt
988.3 ms ✓ Unitful → ConstructionBaseUnitfulExt
11042.6 ms ✓ Automa
1592.0 ms ✓ Interpolations → InterpolationsUnitfulExt
10141.3 ms ✓ PlotUtils
17867.6 ms ✓ ImageCore
2032.8 ms ✓ ImageBase
2570.4 ms ✓ WebP
3462.2 ms ✓ PNGFiles
3650.4 ms ✓ JpegTurbo
2135.1 ms ✓ ImageAxes
4633.4 ms ✓ Sixel
1409.2 ms ✓ ImageMetadata
12760.6 ms ✓ MathTeXEngine
1873.4 ms ✓ Netpbm
50174.9 ms ✓ TiffImages
1199.9 ms ✓ ImageIO
114114.5 ms ✓ Makie
75265.2 ms ✓ CairoMakie
145 dependencies successfully precompiled in 255 seconds. 124 already precompiled.
Precompiling SparseMatrixColoringsColorsExt...
869.0 ms ✓ SparseMatrixColorings → SparseMatrixColoringsColorsExt
1 dependency successfully precompiled in 1 seconds. 29 already precompiled.
Precompiling ZygoteColorsExt...
1720.1 ms ✓ Zygote → ZygoteColorsExt
1 dependency successfully precompiled in 2 seconds. 89 already precompiled.
Precompiling AccessorsIntervalSetsExt...
734.5 ms ✓ Accessors → AccessorsIntervalSetsExt
1 dependency successfully precompiled in 1 seconds. 12 already precompiled.
Precompiling IntervalSetsRecipesBaseExt...
522.3 ms ✓ IntervalSets → IntervalSetsRecipesBaseExt
1 dependency successfully precompiled in 1 seconds. 9 already precompiled.
Precompiling AccessorsUnitfulExt...
603.3 ms ✓ Accessors → AccessorsUnitfulExt
1 dependency successfully precompiled in 1 seconds. 17 already precompiled.
Precompiling DiffEqBaseUnitfulExt...
1562.7 ms ✓ DiffEqBase → DiffEqBaseUnitfulExt
1 dependency successfully precompiled in 2 seconds. 123 already precompiled.
Precompiling NNlibFFTWExt...
860.2 ms ✓ NNlib → NNlibFFTWExt
1 dependency successfully precompiled in 1 seconds. 54 already precompiled.
Precompiling IntervalArithmeticForwardDiffExt...
475.1 ms ✓ IntervalArithmetic → IntervalArithmeticDiffRulesExt
668.3 ms ✓ IntervalArithmetic → IntervalArithmeticForwardDiffExt
2 dependencies successfully precompiled in 1 seconds. 42 already precompiled.
Precompiling IntervalArithmeticRecipesBaseExt...
796.0 ms ✓ IntervalArithmetic → IntervalArithmeticRecipesBaseExt
1 dependency successfully precompiled in 1 seconds. 31 already precompiled.
Precompiling SciMLBaseMakieExt...
9316.1 ms ✓ SciMLBase → SciMLBaseMakieExt
1 dependency successfully precompiled in 10 seconds. 303 already precompiled.Define some Utility Functions
Tip
This section can be skipped. It defines functions to simulate the model, however, from a scientific machine learning perspective, isn't super relevant.
We need a very crude 2-body path. Assume the 1-body motion is a newtonian 2-body position vector
julia
function one2two(path, m₁, m₂)
M = m₁ + m₂
r₁ = m₂ / M .* path
r₂ = -m₁ / M .* path
return r₁, r₂
endone2two (generic function with 1 method)Next we define a function to perform the change of variables:
julia
@views function soln2orbit(soln, model_params=nothing)
@assert size(soln, 1) ∈ [2, 4] "size(soln,1) must be either 2 or 4"
if size(soln, 1) == 2
χ = soln[1, :]
ϕ = soln[2, :]
@assert length(model_params)==3 "model_params must have length 3 when size(soln,2) = 2"
p, M, e = model_params
else
χ = soln[1, :]
ϕ = soln[2, :]
p = soln[3, :]
e = soln[4, :]
end
r = p ./ (1 .+ e .* cos.(χ))
x = r .* cos.(ϕ)
y = r .* sin.(ϕ)
orbit = vcat(x', y')
return orbit
endsoln2orbit (generic function with 2 methods)This function uses second-order one-sided difference stencils at the endpoints; see https://doi.org/10.1090/S0025-5718-1988-0935077-0
julia
function d_dt(v::AbstractVector, dt)
a = -3 / 2 * v[1] + 2 * v[2] - 1 / 2 * v[3]
b = (v[3:end] .- v[1:(end - 2)]) / 2
c = 3 / 2 * v[end] - 2 * v[end - 1] + 1 / 2 * v[end - 2]
return [a; b; c] / dt
endd_dt (generic function with 1 method)This function uses second-order one-sided difference stencils at the endpoints; see https://doi.org/10.1090/S0025-5718-1988-0935077-0
julia
function d2_dt2(v::AbstractVector, dt)
a = 2 * v[1] - 5 * v[2] + 4 * v[3] - v[4]
b = v[1:(end - 2)] .- 2 * v[2:(end - 1)] .+ v[3:end]
c = 2 * v[end] - 5 * v[end - 1] + 4 * v[end - 2] - v[end - 3]
return [a; b; c] / (dt^2)
endd2_dt2 (generic function with 1 method)Now we define a function to compute the trace-free moment tensor from the orbit
julia
function orbit2tensor(orbit, component, mass=1)
x = orbit[1, :]
y = orbit[2, :]
Ixx = x .^ 2
Iyy = y .^ 2
Ixy = x .* y
trace = Ixx .+ Iyy
if component[1] == 1 && component[2] == 1
tmp = Ixx .- trace ./ 3
elseif component[1] == 2 && component[2] == 2
tmp = Iyy .- trace ./ 3
else
tmp = Ixy
end
return mass .* tmp
end
function h_22_quadrupole_components(dt, orbit, component, mass=1)
mtensor = orbit2tensor(orbit, component, mass)
mtensor_ddot = d2_dt2(mtensor, dt)
return 2 * mtensor_ddot
end
function h_22_quadrupole(dt, orbit, mass=1)
h11 = h_22_quadrupole_components(dt, orbit, (1, 1), mass)
h22 = h_22_quadrupole_components(dt, orbit, (2, 2), mass)
h12 = h_22_quadrupole_components(dt, orbit, (1, 2), mass)
return h11, h12, h22
end
function h_22_strain_one_body(dt::T, orbit) where {T}
h11, h12, h22 = h_22_quadrupole(dt, orbit)
h₊ = h11 - h22
hₓ = T(2) * h12
scaling_const = √(T(π) / 5)
return scaling_const * h₊, -scaling_const * hₓ
end
function h_22_quadrupole_two_body(dt, orbit1, mass1, orbit2, mass2)
h11_1, h12_1, h22_1 = h_22_quadrupole(dt, orbit1, mass1)
h11_2, h12_2, h22_2 = h_22_quadrupole(dt, orbit2, mass2)
h11 = h11_1 + h11_2
h12 = h12_1 + h12_2
h22 = h22_1 + h22_2
return h11, h12, h22
end
function h_22_strain_two_body(dt::T, orbit1, mass1, orbit2, mass2) where {T}
# compute (2,2) mode strain from orbits of BH 1 of mass1 and BH2 of mass 2
@assert abs(mass1 + mass2 - 1.0)<1e-12 "Masses do not sum to unity"
h11, h12, h22 = h_22_quadrupole_two_body(dt, orbit1, mass1, orbit2, mass2)
h₊ = h11 - h22
hₓ = T(2) * h12
scaling_const = √(T(π) / 5)
return scaling_const * h₊, -scaling_const * hₓ
end
function compute_waveform(dt::T, soln, mass_ratio, model_params=nothing) where {T}
@assert mass_ratio≤1 "mass_ratio must be <= 1"
@assert mass_ratio≥0 "mass_ratio must be non-negative"
orbit = soln2orbit(soln, model_params)
if mass_ratio > 0
m₂ = inv(T(1) + mass_ratio)
m₁ = mass_ratio * m₂
orbit₁, orbit₂ = one2two(orbit, m₁, m₂)
waveform = h_22_strain_two_body(dt, orbit₁, m₁, orbit₂, m₂)
else
waveform = h_22_strain_one_body(dt, orbit)
end
return waveform
endcompute_waveform (generic function with 2 methods)Simulating the True Model
RelativisticOrbitModel defines system of odes which describes motion of point like particle in schwarzschild background, uses
where,
julia
function RelativisticOrbitModel(u, (p, M, e), t)
χ, ϕ = u
numer = (p - 2 - 2 * e * cos(χ)) * (1 + e * cos(χ))^2
denom = sqrt((p - 2)^2 - 4 * e^2)
χ̇ = numer * sqrt(p - 6 - 2 * e * cos(χ)) / (M * (p^2) * denom)
ϕ̇ = numer / (M * (p^(3 / 2)) * denom)
return [χ̇, ϕ̇]
end
mass_ratio = 0.0 # test particle
u0 = Float64[π, 0.0] # initial conditions
datasize = 250
tspan = (0.0f0, 6.0f4) # timespace for GW waveform
tsteps = range(tspan[1], tspan[2]; length=datasize) # time at each timestep
dt_data = tsteps[2] - tsteps[1]
dt = 100.0
const ode_model_params = [100.0, 1.0, 0.5]; # p, M, eLet's simulate the true model and plot the results using OrdinaryDiffEq.jl
julia
prob = ODEProblem(RelativisticOrbitModel, u0, tspan, ode_model_params)
soln = Array(solve(prob, RK4(); saveat=tsteps, dt, adaptive=false))
waveform = first(compute_waveform(dt_data, soln, mass_ratio, ode_model_params))
begin
fig = Figure()
ax = CairoMakie.Axis(fig[1, 1]; xlabel="Time", ylabel="Waveform")
l = lines!(ax, tsteps, waveform; linewidth=2, alpha=0.75)
s = scatter!(ax, tsteps, waveform; marker=:circle, markersize=12, alpha=0.5)
axislegend(ax, [[l, s]], ["Waveform Data"])
fig
end
Defiing a Neural Network Model
Next, we define the neural network model that takes 1 input (time) and has two outputs. We'll make a function ODE_model that takes the initial conditions, neural network parameters and a time as inputs and returns the derivatives.
It is typically never recommended to use globals but incase you do use them, make sure to mark them as const.
We will deviate from the standard Neural Network initialization and use WeightInitializers.jl,
julia
const nn = Chain(Base.Fix1(fast_activation, cos),
Dense(1 => 32, cos; init_weight=truncated_normal(; std=1e-4), init_bias=zeros32),
Dense(32 => 32, cos; init_weight=truncated_normal(; std=1e-4), init_bias=zeros32),
Dense(32 => 2; init_weight=truncated_normal(; std=1e-4), init_bias=zeros32))
ps, st = Lux.setup(Random.default_rng(), nn)((layer_1 = NamedTuple(), layer_2 = (weight = Float32[-0.00014602141; -3.1497596f-6; 7.7277036f-5; -0.00017809037; -0.00017192417; 0.00018879719; -9.766824f-5; -5.280449f-5; -2.9139488f-5; 5.8075922f-5; -0.00010945252; 9.557834f-5; 0.00022025916; -1.2070078f-5; -3.994704f-5; 7.6106764f-5; 3.867697f-5; -6.3265135f-5; -1.1694666f-5; 0.00011296194; 1.3497404f-7; -5.2378906f-5; 7.237351f-5; 1.905237f-5; -0.00015687192; -0.0001155476; 4.4938723f-5; 0.00013069276; -6.823549f-5; -0.00013750889; 6.580784f-5; -3.4650544f-5;;], bias = Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]), layer_3 = (weight = Float32[9.017568f-5 -0.00011983704 -0.00012524931 5.5220673f-5 1.6906375f-5 6.7001805f-5 -0.00011397639 4.279412f-5 -0.00021391206 -2.8409291f-5 -1.8541834f-6 -3.2764096f-5 7.3001276f-5 -1.7243017f-5 -8.737892f-5 8.627797f-6 -3.23371f-5 2.3803232f-5 -4.6905378f-5 8.613897f-5 5.1724037f-6 0.000111096386 0.00023574695 -6.450389f-5 0.00013492971 -8.267993f-5 -4.6842128f-5 0.0001439572 7.7007934f-7 -4.0448776f-5 4.860206f-5 7.256372f-5; -9.9175944f-5 3.1022844f-6 0.00015226087 -2.0799242f-5 5.6956665f-6 3.0741925f-5 5.423801f-5 -7.6648044f-5 -7.0474896f-5 -0.00013483294 -5.202851f-5 0.0001241278 0.00019244131 9.405456f-5 -6.8661006f-6 9.2927134f-5 0.00020089332 0.0001425216 1.89919f-5 -3.3774857f-5 4.5184395f-5 1.4252557f-5 0.00016825642 -8.2130144f-5 -0.00013865737 -5.9210204f-5 -3.5832316f-5 4.578359f-6 -0.00014783359 9.624002f-5 4.4060027f-5 0.00011399542; -3.2172775f-5 -1.3379678f-5 -1.3149729f-5 -4.3887205f-5 0.00013361483 4.563029f-5 1.3546155f-5 -6.771304f-5 -0.00011363314 -0.00013039485 -2.0425749f-5 -7.441098f-5 9.881837f-5 -0.00022776368 -2.0992291f-5 0.00013343936 -1.141506f-5 -5.972884f-6 -2.293381f-5 0.000103331 -6.5263657f-6 -4.9129787f-5 -7.567483f-5 -9.080154f-5 -8.8772045f-5 3.7454858f-5 -4.603812f-5 2.9996743f-5 -9.661054f-5 0.00015401596 -1.0265582f-5 -4.784908f-5; 0.00017247602 -0.00011626492 -0.00015463016 5.9441216f-5 0.00012115162 1.6090518f-5 -3.843848f-5 -0.00011820016 0.00016854983 6.646964f-5 0.0002394487 8.2820276f-5 0.00011550071 -0.00017997 -9.589719f-5 -0.00017058176 -6.404764f-5 0.00011486406 -9.6579264f-5 3.4403423f-5 -0.00016094197 6.2123334f-5 8.629687f-5 0.00010100329 5.1105922f-5 -1.150412f-5 -8.717629f-5 -2.0124922f-5 -2.3826318f-5 -6.453464f-5 -4.6542624f-5 -0.00011246501; 2.1449869f-6 6.7493136f-5 -2.2987224f-5 3.8307085f-6 -9.731683f-5 -3.5695502f-5 0.00012852083 -0.00010585373 4.33072f-5 -0.00014090199 -7.621631f-5 8.863981f-5 8.008748f-5 6.9718044f-5 8.898034f-6 -2.4834624f-5 -0.00023245116 -0.00013835207 0.00010806997 -1.9611662f-5 -6.5390384f-5 0.00012444428 -9.529073f-5 -0.00024061392 -0.000104452 -2.6980008f-5 -4.4944005f-5 -0.00013536686 -6.856029f-5 -9.8052f-5 -0.0001483173 5.4733417f-5; 9.333313f-5 1.0854802f-5 -0.00016989559 5.434016f-5 9.123863f-5 8.635285f-5 -0.00016929615 1.5645366f-5 1.23193995f-5 7.2008035f-5 7.467736f-5 -2.1675603f-5 4.5122135f-5 -1.3905801f-5 7.7036864f-5 0.0002507801 4.5777753f-5 -0.00010263164 -6.443865f-5 -4.007771f-5 -5.766938f-5 -6.733361f-7 -4.089203f-5 2.3602423f-5 5.2728417f-5 -4.9990525f-5 -4.8926588f-5 -1.6037653f-5 6.811169f-5 -0.00020338043 9.088729f-5 0.00010588634; 1.6260088f-5 1.4607368f-5 2.1722684f-5 -0.00010739381 3.7499856f-5 9.77588f-5 -0.00017595923 0.00011783092 3.9240706f-5 1.3607708f-5 7.858902f-5 6.3328254f-5 -0.000191162 -0.00016412264 -9.4988995f-5 -4.456991f-6 7.492842f-5 -1.527085f-5 -9.74444f-5 0.00021436553 0.00014106363 -4.657296f-5 9.6220516f-5 -2.8235543f-5 -5.820671f-5 4.6024852f-5 -0.00014162398 0.00029088862 -3.8621496f-5 -0.00012191072 -7.577848f-5 4.96738f-5; -2.7631912f-5 -0.00015979348 5.370569f-5 0.00011851289 0.00018058048 -3.0751085f-5 6.113319f-5 3.5265817f-5 0.00016009998 -0.00011860091 4.2198382f-5 -1.6208967f-5 0.00020594154 0.00012372529 9.1883565f-5 7.9399964f-5 -2.0186733f-5 -6.124558f-5 0.00013630207 -1.8109322f-5 -0.00020225038 1.6420347f-5 -0.00021858363 -0.00015900313 -3.3306293f-5 0.00017877758 -2.111228f-5 0.00013957566 -2.927258f-5 2.5667096f-5 -8.0766076f-5 -0.00011159258; -0.00015525125 -0.00015034681 -7.1275164f-5 -9.494937f-6 2.346547f-5 2.9075503f-5 -5.773935f-5 -9.259039f-5 -9.769226f-5 -0.00015063157 -8.997084f-6 0.00011056838 7.401127f-6 -0.00018384383 9.6623946f-5 0.00014986598 0.00013839464 -0.00019822232 4.546336f-5 0.00019935433 -1.2524061f-5 7.5010445f-5 4.0642073f-5 -0.00010471576 -4.6528356f-5 -0.00010817473 1.899666f-5 -7.928422f-5 -1.9331142f-6 9.246857f-5 -5.2552434f-5 5.4596658f-5; -4.9021655f-5 -7.716848f-5 0.000117677126 7.632552f-6 3.903397f-5 -0.000100972065 0.00015059162 6.430632f-5 -0.00011422798 -0.00010925679 2.8010678f-5 -0.00013655501 4.6315126f-5 -8.843219f-5 0.000100149264 -0.00012476015 2.3678353f-5 1.2219975f-7 -1.0801165f-5 2.6921402f-6 -1.9605812f-5 -0.000110082095 -1.9724683f-5 -3.86767f-5 3.7696205f-5 6.5467495f-5 -1.5082701f-5 -3.307644f-5 5.4028566f-5 -6.1008443f-5 -8.6831795f-5 0.00017622048; 6.885172f-5 3.0491158f-6 -0.00012858245 7.438504f-5 -0.00013262498 -7.925293f-5 9.259963f-5 -1.0405765f-6 -5.6645687f-5 -5.254046f-6 5.3157182f-5 4.7816582f-5 5.0831826f-5 -4.573917f-6 -5.7084053f-5 -0.000105326355 -7.087354f-5 0.00013732997 -9.557527f-5 0.00011672151 0.00016990023 6.120259f-6 4.698611f-6 1.1892598f-5 9.386885f-5 -4.7510086f-5 6.858021f-5 -0.00011458535 2.693269f-5 -2.2665308f-5 -0.000117370946 0.00012456618; 2.8703418f-5 -0.00011705616 3.9256458f-5 7.5689255f-5 -2.3586736f-5 -8.872946f-5 0.0001276579 -7.028031f-5 4.009923f-5 -0.00021085063 -5.06383f-5 -0.000114453214 4.3945503f-5 -6.882643f-5 9.116355f-5 2.7077282f-5 -0.00014491507 3.1683772f-5 -0.00015451397 -0.00012648024 0.00014341857 1.20918685f-5 -0.00013324854 0.0001678058 1.0425042f-5 0.0001041631 9.5511336f-5 -4.097354f-6 -0.0001660044 1.37386105f-5 0.00015473782 6.519558f-5; -8.104525f-5 1.569785f-5 8.744833f-5 0.00014225017 -0.00020580801 4.3222895f-5 -8.1609265f-5 0.00017734904 5.763818f-6 -0.000181178 2.8269298f-5 -2.5930902f-5 0.00015603089 -0.000121466175 -0.00016421876 -6.9229514f-5 0.00027272778 -9.841526f-5 8.211068f-5 0.00014113149 0.00016239959 -7.5550764f-5 9.950333f-5 0.0001457159 0.00010193034 0.00019880259 2.2393303f-6 2.0675669f-5 -6.499599f-5 -3.9694172f-5 8.8626875f-5 2.3063812f-5; -9.974397f-5 0.00022023331 -6.9246584f-5 -0.00017861248 -4.6205165f-5 9.44442f-5 0.00011381958 8.0691774f-5 -7.308611f-5 0.0002286192 1.39368185f-5 7.4704694f-5 -2.586936f-6 5.646986f-5 -6.0292397f-5 -8.226658f-5 4.275077f-5 5.6420053f-5 -0.00013487831 0.00013401327 4.511038f-5 1.1059833f-5 -4.370535f-6 8.2337225f-5 2.4992512f-6 -7.987191f-5 -5.1331776f-6 -0.00013680304 2.2207629f-5 -0.00013935115 2.0946778f-5 -0.00013993488; 4.222857f-5 -8.002966f-5 3.898926f-5 -7.827437f-5 0.00012556673 -6.713673f-5 6.997405f-5 -8.242168f-6 0.000116645824 3.2174412f-5 0.00024399138 -4.474084f-5 -1.5302172f-5 -5.8030302f-5 -8.190601f-5 6.876208f-5 -2.0332547f-5 -1.53497f-5 0.0001008791 -2.786875f-5 -3.7251077f-5 3.0945233f-5 -7.8787016f-5 -5.2648684f-5 1.4670433f-5 -9.520105f-5 -4.8563925f-5 -0.00015644704 -5.5258886f-5 -0.00013230376 -0.00014161119 -2.1476128f-5; 5.4488115f-5 0.00013114855 -8.892809f-5 -2.5868236f-5 -2.2094242f-5 -4.356766f-5 -6.1814164f-5 1.2769882f-5 -1.8768029f-5 -0.000121704645 -3.5330372f-6 -5.5333723f-5 1.7948112f-5 2.2111817f-5 8.9488996f-5 8.134761f-5 -4.606828f-5 4.9316925f-5 -1.3175137f-5 -0.00017562929 5.29097f-5 8.845565f-6 0.00019758953 6.8594534f-5 -0.00011511197 -4.8692524f-5 2.9816834f-5 -4.795355f-5 -8.991001f-5 -4.0034938f-5 -0.00018729473 -2.6388354f-5; 0.00013391789 3.2754448f-5 -5.6603472f-5 7.8653364f-5 0.00013219327 -0.00011722584 -9.038759f-5 4.7423142f-5 -5.830967f-5 3.696669f-5 0.00012996403 -0.00013932389 5.186701f-5 0.00011294823 -9.110602f-5 -0.00010449767 -0.00016678747 -7.580733f-5 -7.442823f-5 -2.7453f-5 -7.379238f-5 -5.159694f-5 -2.0301126f-5 0.000101109596 0.00010861042 0.00020374138 2.1445429f-5 0.000115675626 0.00026261888 5.821783f-5 -2.1264677f-5 -7.173909f-5; 9.5507985f-6 -2.9205512f-5 -0.00022115113 -6.0469243f-5 -4.2123578f-5 -0.000106563486 -9.648688f-5 -8.730864f-5 1.915841f-5 0.00010323437 0.00012881667 5.916112f-5 0.00015780612 -0.000119360375 -1.8127002f-5 -2.2588474f-6 -1.3990197f-5 3.032461f-5 -2.6321746f-5 -5.2933647f-6 0.00011406521 2.8557337f-5 5.156117f-5 -2.8535567f-5 8.3259323f-7 -0.0001834655 -2.3134486f-5 2.4380357f-5 0.00014296758 -0.00010418675 8.853499f-6 0.00011036525; 1.8213945f-5 -3.7777568f-7 -8.513244f-5 7.081781f-5 5.3873705f-6 -2.3891653f-6 8.334284f-5 2.2332359f-5 0.00010174954 9.27013f-5 -2.5864623f-5 0.00013861227 0.0001552034 0.00013561564 -8.526065f-6 2.1668093f-5 0.00017367859 3.3049473f-5 0.00011464365 -1.9867159f-5 0.00010191426 -3.993447f-5 0.00017675626 8.0364116f-5 -3.8759248f-5 4.76716f-6 0.00011005 -3.9345774f-5 9.413417f-6 -4.0006682f-5 6.823275f-5 -2.0566362f-5; -0.00018025361 9.847612f-5 -0.00017573667 6.149217f-5 2.8071672f-6 -1.1264357f-5 0.0001272266 -6.447605f-5 -1.5773092f-5 0.00014308008 9.426951f-5 6.468488f-5 8.658613f-5 0.00012923643 -2.1931819f-5 0.00012721299 -5.9459602f-5 0.00010812791 -2.2162898f-5 2.8201324f-5 1.647285f-5 3.4140343f-5 -5.55986f-5 -2.8278273f-5 -6.958834f-5 9.477071f-5 0.00016174688 -2.733396f-5 -3.2339128f-7 4.746204f-5 -9.861674f-5 -0.00014600392; -0.00010364952 1.2420151f-5 9.82755f-5 -5.3507465f-5 0.00013771413 4.7838108f-5 0.00017362731 -0.00016233342 -0.0001184562 0.0001049214 0.00011248104 3.823775f-5 -3.914222f-5 -7.8632234f-5 -0.00014317222 5.2474934f-5 0.00016349053 1.4013536f-5 -0.00012549886 7.896745f-5 -0.00011729475 0.00022082007 -1.9210247f-6 5.9661106f-5 -0.0002526959 -0.00016572108 2.8099443f-5 -9.599564f-6 -9.909724f-5 -7.441488f-6 1.756754f-5 -0.00031461223; 4.7709564f-5 -3.069263f-5 -0.00011312365 -4.2373475f-5 -1.60711f-5 -6.326767f-5 5.6841713f-5 1.2907916f-5 2.648332f-5 -3.089792f-5 5.848508f-5 -2.7402846f-5 6.7960915f-5 7.51231f-6 0.00010614698 2.9167157f-5 -0.00014495257 -0.00014660657 -2.062536f-5 0.00017890388 2.2887503f-5 1.6941141f-5 -5.48596f-5 9.9903584f-5 -0.00015841995 0.00013626878 -7.860564f-5 5.851919f-5 0.0002130086 5.0514853f-5 -7.7934856f-5 -2.4478875f-5; -8.66265f-6 -4.9235427f-5 5.2978547f-5 -1.0049968f-5 5.157117f-5 -3.183429f-5 -0.00014209848 2.6873735f-5 3.23743f-5 2.3756014f-5 2.387835f-5 0.0001593284 4.9579103f-5 -0.00011273541 -5.442689f-5 4.9436625f-5 -0.00013564194 0.00015542321 -5.3759803f-5 0.00013734488 -8.262759f-5 -4.0436338f-5 -5.65094f-5 5.3608845f-5 -8.578812f-5 -3.761377f-5 9.80683f-5 -3.0766405f-5 -3.191407f-5 0.00024312985 8.7276894f-5 -0.000118397824; 7.706472f-7 -1.2820788f-6 3.8107493f-5 -0.00015184055 -5.193691f-5 -0.00012245754 5.4232096f-5 0.0001613332 0.00013219709 2.8858603f-5 -0.00013345963 7.829147f-5 0.00012828984 -5.7910032f-8 9.664839f-5 -2.5425925f-6 8.87399f-5 -7.280643f-5 -9.8650686f-5 -5.577383f-5 9.632638f-5 -9.4652496f-5 5.3162014f-5 -5.8901885f-5 -0.0002007483 -3.8354232f-5 3.275669f-5 1.0960447f-5 9.677559f-5 -0.0001338144 0.00016542472 1.5367976f-6; 0.00026350116 0.00013037518 7.5729076f-5 -0.00013175598 -3.186532f-5 -5.5466127f-5 -4.8181806f-5 0.00027214971 0.00017891619 1.1925176f-6 0.00013493278 -0.0001255698 -1.1964418f-5 6.464445f-5 -0.0001488489 9.7079894f-5 7.478813f-5 -3.5884415f-5 -7.162318f-5 2.2470042f-5 -0.00013397243 0.0001667365 -5.177735f-5 -2.4379306f-6 0.0001531192 -7.4759584f-5 0.00018474099 -4.836645f-5 0.0001494476 7.156383f-5 3.761533f-5 8.2044906f-5; -4.969204f-5 -0.00014198422 -1.1563982f-5 -4.8356877f-5 1.583191f-5 -0.00015769611 -4.0777773f-5 -3.9769904f-5 -3.4433637f-5 6.115869f-5 -7.075846f-5 -3.707532f-5 4.3174296f-5 -7.3216645f-5 -0.00013633951 3.2403161f-6 0.00016164752 -0.00011728008 1.5372545f-5 3.861531f-5 -8.3707244f-5 4.0025796f-5 -6.277678f-5 -0.00016161137 8.224036f-5 -2.7148253f-5 0.00012718143 4.9897746f-5 -0.00013978324 -1.3813678f-5 -3.7191083f-5 -8.610853f-5; 9.619317f-5 -7.6401026f-5 5.7173806f-6 4.3761538f-5 -0.00018580545 2.1234406f-5 1.1352531f-5 2.2311573f-5 3.0859483f-5 0.0001457476 2.9466664f-6 0.00019010429 0.00011393038 -0.00011875788 3.3461452f-5 -0.000111203066 3.1546904f-5 -8.693212f-5 -0.00023879865 3.3661403f-5 -0.00011831856 -0.000113353286 -0.00023989972 8.550974f-6 -0.00013630238 5.58231f-7 4.0505012f-5 -4.417f-5 5.8078326f-6 4.1773805f-5 3.6905087f-5 9.268829f-5; -7.1494906f-5 -0.00012945852 -0.00013669218 -1.34214515f-5 -1.2071266f-5 -0.00012070796 -2.4554465f-6 7.879436f-5 0.00022498415 -5.874636f-5 3.059564f-6 -0.00017994663 7.272161f-5 5.1819774f-5 -0.00011399122 9.175459f-5 2.5800748f-5 -5.0135855f-5 -1.0376702f-5 9.285134f-5 3.0677552f-6 -6.1977924f-5 9.2496575f-6 -0.00012419623 -0.0001158796 0.00011968428 9.138096f-5 1.8154782f-5 4.8115206f-5 -6.151662f-5 9.677626f-5 2.5361064f-6; 3.0575964f-5 -0.0001907324 4.6703717f-5 4.6176123f-5 7.126713f-5 -0.00017414308 1.8643535f-5 1.4234583f-5 6.3018044f-5 -2.0136235f-5 -0.00012290479 0.00013266108 1.26563145f-5 6.477549f-5 -6.6109526f-5 -0.00017350647 -9.5886004f-5 -0.00016564256 -0.00014114144 1.7436503f-5 0.00014731813 3.6338883f-5 -0.00013743268 0.00012845326 -0.000118164935 2.5955749f-6 0.00023234703 0.00017074932 4.751271f-5 -1.8884931f-6 0.00029555245 6.656346f-5; 9.092418f-5 -7.199893f-6 -2.8965685f-5 5.8971355f-6 -1.3236706f-5 2.2599272f-7 -6.3832344f-6 1.5962689f-6 2.8019973f-5 -0.00010381884 2.7633228f-5 2.0019275f-5 3.8022004f-5 9.670923f-5 1.575427f-5 1.2839249f-5 -7.902891f-5 -2.6517399f-5 0.0001591945 8.4465646f-5 0.00010128189 0.00020205855 -3.8554103f-5 -9.7900565f-6 -3.4876633f-5 8.115763f-5 -7.866772f-5 -6.05329f-5 -8.283124f-5 7.0528476f-6 3.0447918f-5 -4.418855f-6; 1.5738116f-5 -5.919915f-5 -5.937516f-6 -3.5382018f-5 8.6241445f-5 6.664042f-5 -7.179568f-5 -0.00014423761 -3.6184014f-5 2.1955053f-5 -2.3974746f-6 0.00026438967 -3.466614f-5 0.00014532801 -0.00016021609 4.564213f-5 -1.3742485f-5 7.139149f-5 0.00016314638 -7.255433f-5 5.7274367f-5 -1.8364703f-5 2.4807754f-5 -0.00012924949 -7.222538f-5 -5.1219424f-5 1.3436846f-6 -7.115517f-6 -0.00014999961 8.3853716f-5 -0.0003222688 0.00015827737; 0.00014062237 -0.00024629378 4.981746f-5 0.00011812783 -2.1343114f-5 0.00014485611 -1.4763909f-5 -0.00013871175 -8.3736915f-5 7.159475f-5 -0.00013272808 9.65489f-5 -9.27801f-5 0.000101006495 0.00014597579 -8.932649f-5 -0.0001867751 -1.4267136f-5 -2.0163376f-5 4.7296f-5 -0.00014221745 3.6256526f-5 -2.0418518f-5 9.05062f-5 -0.00010670207 -6.5138793f-6 -0.00013926378 7.727058f-5 8.491684f-5 -0.000113522285 -5.5775297f-5 -0.00018441405], bias = Float32[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]), layer_4 = (weight = Float32[8.28411f-5 -9.906505f-5 8.458642f-5 -0.00027114083 -3.2896014f-6 3.1335174f-5 -5.545565f-5 0.00012917617 0.00015331886 3.5532998f-5 -7.103476f-5 -1.8354536f-5 -5.4835764f-5 -0.00011879689 -1.3539128f-5 -0.00013843352 6.97117f-5 -8.2598424f-5 -0.00019591153 -0.00012397394 6.6949106f-5 4.8665565f-6 -2.7792677f-5 -0.00019137943 -3.31835f-5 0.000113985036 6.593956f-5 -0.00013732683 7.680594f-6 4.3998905f-5 0.000103844155 7.969053f-6; 0.00021923706 8.316737f-5 -6.3142514f-5 -0.00016211328 -6.671226f-5 -9.9439625f-5 -3.5558788f-8 -0.000116534655 5.7756086f-5 0.00018383168 8.217614f-5 -7.5852535f-5 5.6479594f-6 3.7273036f-5 -9.336621f-5 6.165728f-5 -0.00017774168 -0.00015634266 6.9851376f-5 1.2096053f-5 -7.233856f-5 7.216045f-5 -8.609886f-5 4.543341f-5 2.0768477f-5 5.2457857f-5 4.8057227f-5 0.00024547643 -0.00016561113 -9.586822f-6 7.346302f-5 -2.4387946f-5], bias = Float32[0.0, 0.0])), (layer_1 = NamedTuple(), layer_2 = NamedTuple(), layer_3 = NamedTuple(), layer_4 = NamedTuple()))Similar to most DL frameworks, Lux defaults to using Float32, however, in this case we need Float64
julia
const params = ComponentArray(ps |> f64)
const nn_model = StatefulLuxLayer{true}(nn, nothing, st)StatefulLuxLayer{true}(
Chain(
layer_1 = WrappedFunction(Base.Fix1{typeof(LuxLib.API.fast_activation), typeof(cos)}(LuxLib.API.fast_activation, cos)),
layer_2 = Dense(1 => 32, cos), # 64 parameters
layer_3 = Dense(32 => 32, cos), # 1_056 parameters
layer_4 = Dense(32 => 2), # 66 parameters
),
) # Total: 1_186 parameters,
# plus 0 states.Now we define a system of odes which describes motion of point like particle with Newtonian physics, uses
where,
julia
function ODE_model(u, nn_params, t)
χ, ϕ = u
p, M, e = ode_model_params
# In this example we know that `st` is am empty NamedTuple hence we can safely ignore
# it, however, in general, we should use `st` to store the state of the neural network.
y = 1 .+ nn_model([first(u)], nn_params)
numer = (1 + e * cos(χ))^2
denom = M * (p^(3 / 2))
χ̇ = (numer / denom) * y[1]
ϕ̇ = (numer / denom) * y[2]
return [χ̇, ϕ̇]
endODE_model (generic function with 1 method)Let us now simulate the neural network model and plot the results. We'll use the untrained neural network parameters to simulate the model.
julia
prob_nn = ODEProblem(ODE_model, u0, tspan, params)
soln_nn = Array(solve(prob_nn, RK4(); u0, p=params, saveat=tsteps, dt, adaptive=false))
waveform_nn = first(compute_waveform(dt_data, soln_nn, mass_ratio, ode_model_params))
begin
fig = Figure()
ax = CairoMakie.Axis(fig[1, 1]; xlabel="Time", ylabel="Waveform")
l1 = lines!(ax, tsteps, waveform; linewidth=2, alpha=0.75)
s1 = scatter!(
ax, tsteps, waveform; marker=:circle, markersize=12, alpha=0.5, strokewidth=2)
l2 = lines!(ax, tsteps, waveform_nn; linewidth=2, alpha=0.75)
s2 = scatter!(
ax, tsteps, waveform_nn; marker=:circle, markersize=12, alpha=0.5, strokewidth=2)
axislegend(ax, [[l1, s1], [l2, s2]],
["Waveform Data", "Waveform Neural Net (Untrained)"]; position=:lb)
fig
end
Setting Up for Training the Neural Network
Next, we define the objective (loss) function to be minimized when training the neural differential equations.
julia
const mseloss = MSELoss()
function loss(θ)
pred = Array(solve(prob_nn, RK4(); u0, p=θ, saveat=tsteps, dt, adaptive=false))
pred_waveform = first(compute_waveform(dt_data, pred, mass_ratio, ode_model_params))
return mseloss(pred_waveform, waveform)
endloss (generic function with 1 method)Warmup the loss function
julia
loss(params)0.0006908056702303812Now let us define a callback function to store the loss over time
julia
const losses = Float64[]
function callback(θ, l)
push!(losses, l)
@printf "Training \t Iteration: %5d \t Loss: %.10f\n" θ.iter l
return false
endcallback (generic function with 1 method)Training the Neural Network
Training uses the BFGS optimizers. This seems to give good results because the Newtonian model seems to give a very good initial guess
julia
adtype = Optimization.AutoZygote()
optf = Optimization.OptimizationFunction((x, p) -> loss(x), adtype)
optprob = Optimization.OptimizationProblem(optf, params)
res = Optimization.solve(
optprob, BFGS(; initial_stepnorm=0.01, linesearch=LineSearches.BackTracking());
callback, maxiters=1000)retcode: Success
u: ComponentVector{Float64}(layer_1 = Float64[], layer_2 = (weight = [-0.00014602141163767257; -3.1497595500658038e-6; 7.727703632553489e-5; -0.00017809036944518703; -0.00017192416999025928; 0.00018879718845707825; -9.766824223338588e-5; -5.2804491133315926e-5; -2.9139488105980368e-5; 5.807592242489027e-5; -0.00010945252142838737; 9.557833982399476e-5; 0.00022025915677638562; -1.2070077900696788e-5; -3.994704093204682e-5; 7.610676402689434e-5; 3.867697159870576e-5; -6.326513539529023e-5; -1.1694665772661983e-5; 0.0001129619413404586; 1.3497404438553112e-7; -5.237890582066515e-5; 7.23735065547817e-5; 1.9052369680109336e-5; -0.00015687191626038657; -0.00011554759839770381; 4.493872256712334e-5; 0.00013069275883019398; -6.823548756066098e-5; -0.00013750889047507055; 6.580784247478982e-5; -3.465054396654612e-5;;], bias = [-1.150160217250942e-16, -4.22914484204598e-18, 6.171874474872738e-18, -9.664665921397983e-17, 2.816782781658119e-17, 2.4755338869891732e-17, 5.241692134749965e-17, -1.283539500194225e-16, -3.510931727008827e-17, 1.7038008316865665e-17, -8.810319229390369e-17, -7.745016928333598e-18, 2.3572456839412352e-16, -2.237389634900018e-17, 3.0393978408672605e-17, 2.6419404798172328e-17, 6.371888231866082e-17, -5.631789370573952e-17, -4.266480917723592e-18, 1.6854903866369633e-17, 1.5919288641847288e-19, 6.042888855735013e-18, 9.758898345485934e-17, 3.381870160100467e-17, -2.077622818026805e-16, -1.3792010651776985e-16, 5.0481266510442424e-17, 7.999646081750175e-17, -9.126771864903048e-17, -8.70953501546409e-17, 1.210902454612978e-16, -1.9553297354174248e-17]), layer_3 = (weight = [9.017627489179605e-5 -0.000119836441815042 -0.00012524871381805678 5.522126847886512e-5 1.690697062536216e-5 6.700240082232235e-5 -0.00011397579620001378 4.279471503928898e-5 -0.00021391146693253258 -2.840869565892849e-5 -1.8535879070834973e-6 -3.2763499991181275e-5 7.300187170862858e-5 -1.7242421751635228e-5 -8.73783221385939e-5 8.628392418411782e-6 -3.233650314263381e-5 2.3803827200989346e-5 -4.6904781999674695e-5 8.613956633149114e-5 5.17299922017562e-6 0.00011109698133901354 0.00023574754485985812 -6.450329347621012e-5 0.00013493030455088412 -8.267933408837377e-5 -4.6841532100134964e-5 0.00014395778888961615 7.706748697803776e-7 -4.0448180886364545e-5 4.860265725534042e-5 7.25643174455936e-5; -9.917365126284126e-5 3.1045774608370653e-6 0.00015226315851352113 -2.0796948531373826e-5 5.6979595387904315e-6 3.074421808150167e-5 5.424030429595749e-5 -7.664575126225886e-5 -7.047260326769815e-5 -0.00013483064570435192 -5.202621717443275e-5 0.00012413008632166942 0.00019244360125183593 9.405685092817431e-5 -6.8638075685882235e-6 9.292942674393715e-5 0.00020089561555098083 0.00014252389634522691 1.8994192275860352e-5 -3.377256392240169e-5 4.5186688355462736e-5 1.425484994571833e-5 0.00016825871104403386 -8.212785063049402e-5 -0.00013865507820128066 -5.920791111974665e-5 -3.583002283674433e-5 4.5806521188820215e-6 -0.00014783129766554538 9.624231576481732e-5 4.4062320073424476e-5 0.00011399771143944296; -3.217486903393059e-5 -1.3381772646418997e-5 -1.3151823281976307e-5 -4.388929903595353e-5 0.0001336127328386551 4.5628197199252124e-5 1.3544060834660371e-5 -6.771513164948413e-5 -0.00011363523537965363 -0.00013039694624693762 -2.0427842741795385e-5 -7.44130726388773e-5 9.881627318013088e-5 -0.0002277657755484983 -2.0994385181430807e-5 0.00013343726584481532 -1.141715404172955e-5 -5.974978307667272e-6 -2.2935903530189902e-5 0.00010332890940391461 -6.528459948633563e-6 -4.91318811555088e-5 -7.56769210283788e-5 -9.08036369082695e-5 -8.877413949901365e-5 3.745276387691541e-5 -4.604021398789974e-5 2.9994649025884615e-5 -9.661263777615118e-5 0.000154013863152633 -1.0267676429966092e-5 -4.78511743722189e-5; 0.00017247577578513937 -0.00011626516699339383 -0.00015463041110211126 5.944096936714041e-5 0.00012115137450150333 1.6090271842993615e-5 -3.843872674638448e-5 -0.00011820040437174126 0.00016854958174503294 6.646939526530821e-5 0.00023944844814236256 8.282002907930827e-5 0.00011550046564066124 -0.00017997024316587605 -9.589743332485776e-5 -0.00017058200229671858 -6.404788929517487e-5 0.00011486381207345899 -9.657951069544188e-5 3.440317685585453e-5 -0.0001609422170209245 6.212308743256047e-5 8.629662714651708e-5 0.00010100304005886312 5.1105675980076994e-5 -1.1504366752482129e-5 -8.717653601211599e-5 -2.0125168716325237e-5 -2.382656476998317e-5 -6.453488369020599e-5 -4.654287023600275e-5 -0.00011246525447364073; 2.1407823050904678e-6 6.748893159688316e-5 -2.2991428489184673e-5 3.826503904747066e-6 -9.73210325928481e-5 -3.569970700754716e-5 0.000128516622714384 -0.00010585793548947886 4.33029936511533e-5 -0.00014090619421316922 -7.622051540513887e-5 8.86356074597332e-5 8.008327225709297e-5 6.97138393997669e-5 8.893829578441446e-6 -2.4838828688224766e-5 -0.0002324553633689408 -0.0001383562766596169 0.00010806576850605082 -1.9615866188386246e-5 -6.539458896718124e-5 0.00012444007828613048 -9.52949330558477e-5 -0.00024061812924905767 -0.00010445620770316145 -2.6984212093364364e-5 -4.4948209472790836e-5 -0.00013537106766582137 -6.856449814498859e-5 -9.805620262615735e-5 -0.0001483215009675781 5.472921205814956e-5; 9.333413877456534e-5 1.085580866047259e-5 -0.00016989458267644398 5.434116506065917e-5 9.123963795999811e-5 8.63538542499448e-5 -0.00016929514563244298 1.56463728227209e-5 1.2320406177514737e-5 7.200904187911858e-5 7.467836517307363e-5 -2.1674596745380997e-5 4.512314158190169e-5 -1.3904794422068362e-5 7.703787091586532e-5 0.00025078111762113527 4.5778759932637035e-5 -0.0001026306319080508 -6.443764609584746e-5 -4.0076703179605425e-5 -5.7668371618000475e-5 -6.723294281072988e-7 -4.089102471780357e-5 2.3603429191190015e-5 5.2729424033798034e-5 -4.9989518764872414e-5 -4.892558094879833e-5 -1.6036646365050818e-5 6.811269381714679e-5 -0.00020337942728374921 9.088829651873264e-5 0.00010588734844632252; 1.626027642154503e-5 1.4607556291905098e-5 2.1722872585398112e-5 -0.00010739362278733098 3.750004452407678e-5 9.775898706388009e-5 -0.00017595903864608636 0.0001178311058505114 3.924089378082688e-5 1.36078961043876e-5 7.858921071269044e-5 6.332844258971955e-5 -0.00019116181738687473 -0.00016412245259167128 -9.498880627111974e-5 -4.456802802140614e-6 7.492861095337688e-5 -1.5270662519158706e-5 -9.74442091352629e-5 0.00021436571750777262 0.0001410638205892084 -4.657277259289736e-5 9.622070410509129e-5 -2.8235354878832273e-5 -5.820652313892383e-5 4.602504031603641e-5 -0.0001416237946647543 0.00029088881182937347 -3.8621307797855646e-5 -0.0001219105333836152 -7.5778291833247e-5 4.967398762923226e-5; -2.7630622775412723e-5 -0.0001597921864452009 5.370697997545754e-5 0.00011851418182753751 0.00018058176735916056 -3.0749796700788654e-5 6.113448123482768e-5 3.526710586424979e-5 0.00016010127089953192 -0.00011859962162293456 4.21996710208492e-5 -1.6207678549338285e-5 0.00020594283066723522 0.000123726575998945 9.188485362369678e-5 7.940125303843928e-5 -2.0185444577009196e-5 -6.124428821226571e-5 0.0001363033597733656 -1.8108033167895656e-5 -0.00020224908714612768 1.6421635314827487e-5 -0.00021858234597334678 -0.000159001842825318 -3.3305003875521066e-5 0.00017877887237386405 -2.111099186026863e-5 0.0001395769440712168 -2.927129119075385e-5 2.5668384274233577e-5 -8.076478754833343e-5 -0.00011159128828171512; -0.00015525313316529476 -0.000150348686623976 -7.127704262885124e-5 -9.496815586217386e-6 2.3463592548167246e-5 2.9073625079138096e-5 -5.774122943321816e-5 -9.259226604982331e-5 -9.769413842505946e-5 -0.00015063345305307783 -8.998961824457688e-6 0.00011056649858446192 7.399248566908559e-6 -0.00018384571118432313 9.662206760921389e-5 0.0001498641057297321 0.00013839275819563178 -0.00019822419668699597 4.546148087328086e-5 0.00019935244821326433 -1.2525939531689401e-5 7.5008566305851e-5 4.064019488950355e-5 -0.00010471763486194624 -4.653023385244079e-5 -0.00010817661055980594 1.899478123604221e-5 -7.928609662840257e-5 -1.9349924185994045e-6 9.246668818023051e-5 -5.2554312027412994e-5 5.4594779463595156e-5; -4.902268297332817e-5 -7.716950672507547e-5 0.00011767609798520741 7.631524609726491e-6 3.9032942409951084e-5 -0.00010097309309816916 0.0001505905875510915 6.430528996683489e-5 -0.00011422900748035174 -0.00010925781857982429 2.8009650164718273e-5 -0.0001365560401190626 4.63140986958562e-5 -8.84332202567372e-5 0.0001001482360571043 -0.00012476117440560905 2.367732490351211e-5 1.211720636254077e-7 -1.0802193131856872e-5 2.6911125551541594e-6 -1.9606839445222005e-5 -0.0001100831231760437 -1.9725710402741965e-5 -3.867772643748921e-5 3.769517756704804e-5 6.546646731844356e-5 -1.5083728570933074e-5 -3.307746915557111e-5 5.402753781349292e-5 -6.100947025875188e-5 -8.683282244779286e-5 0.0001762194497551188; 6.885212197293286e-5 3.0495202338845207e-6 -0.00012858205017774404 7.438544408277232e-5 -0.00013262457222818746 -7.925252729702239e-5 9.260003271837634e-5 -1.040172077142585e-6 -5.664528297866664e-5 -5.253641387486162e-6 5.315758672285708e-5 4.781698654032233e-5 5.0832230687126117e-5 -4.5735126093002705e-6 -5.7083648510984865e-5 -0.00010532595051127521 -7.087313433085546e-5 0.00013733037913543161 -9.557486491185574e-5 0.00011672191313327163 0.00016990063587063033 6.1206635166798345e-6 4.699015522686257e-6 1.1893002653136289e-5 9.386925804055011e-5 -4.750968154776977e-5 6.858061233860454e-5 -0.00011458494854533298 2.69330946196474e-5 -2.2664903821085796e-5 -0.00011737054164196381 0.0001245665886984116; 2.8702678752794032e-5 -0.00011705689845529329 3.925571858145497e-5 7.568851564157642e-5 -2.3587474936886473e-5 -8.873019847940671e-5 0.00012765715931619805 -7.028104752224745e-5 4.00984891139258e-5 -0.0002108513699231376 -5.06390396940208e-5 -0.00011445395372264742 4.3944764035607796e-5 -6.882717294773958e-5 9.116281129264368e-5 2.707654224378719e-5 -0.00014491581256423982 3.168303286384945e-5 -0.00015451470695824297 -0.00012648098069165472 0.00014341782938300692 1.2091129098745364e-5 -0.0001332492801911831 0.00016780506397209198 1.0424302706284876e-5 0.00010416235730828165 9.551059614478715e-5 -4.098093507295691e-6 -0.00016600514660511926 1.3737871119715729e-5 0.0001547370802991388 6.519483853208918e-5; -8.104239564468398e-5 1.570070130207449e-5 8.745118215834971e-5 0.00014225301824429722 -0.000205805161707905 4.322574718192506e-5 -8.160641332700929e-5 0.00017735188852917948 5.766669933898655e-6 -0.00018117514296512558 2.8272149980563382e-5 -2.5928049903214414e-5 0.00015603374017320596 -0.00012146332333407505 -0.00016421590440052834 -6.922666248477729e-5 0.00027273062700793814 -9.84124056723298e-5 8.211353237663256e-5 0.00014113433976113417 0.00016240244408060675 -7.55479125965704e-5 9.950618518438508e-5 0.00014571874603085992 0.00010193318998594215 0.00019880544245630066 2.2421821957929145e-6 2.067852041896116e-5 -6.499313927100599e-5 -3.96913204478102e-5 8.862972722078146e-5 2.3066664330933664e-5; -9.974379972734526e-5 0.00022023348392506212 -6.924641238472379e-5 -0.00017861230626183656 -4.620499348564158e-5 9.444437128079005e-5 0.0001138197516422442 8.069194555102913e-5 -7.308593521772807e-5 0.00022861937432137378 1.3936989854504498e-5 7.470486562053702e-5 -2.586764684363892e-6 5.6470031632874976e-5 -6.029222604299822e-5 -8.226641111721238e-5 4.275094070292152e-5 5.642022406502686e-5 -0.0001348781403055686 0.00013401344440464873 4.511055194709532e-5 1.1060004363775448e-5 -4.3703636026982605e-6 8.233739608951924e-5 2.4994225452456333e-6 -7.987174067127512e-5 -5.13300628770189e-6 -0.0001368028639251647 2.2207800067212052e-5 -0.00013935097704122798 2.0946949372250223e-5 -0.00013993471256869768; 4.222698068459143e-5 -8.003125161069987e-5 3.8987669785066946e-5 -7.827595594009223e-5 0.00012556514072328855 -6.7138320201967e-5 6.997245897634834e-5 -8.243757958034319e-6 0.00011664423461167573 3.217282234670226e-5 0.00024398979100494724 -4.4742427900406324e-5 -1.53037615122264e-5 -5.803189172314085e-5 -8.190759738811753e-5 6.876048816463664e-5 -2.033413668998983e-5 -1.5351289886374602e-5 0.00010087751094812924 -2.787033893712216e-5 -3.7252666271734834e-5 3.094364389523462e-5 -7.878860536168601e-5 -5.265027332955564e-5 1.4668843749480127e-5 -9.520264187033145e-5 -4.856551434516414e-5 -0.0001564486335441157 -5.526047582991724e-5 -0.0001323053472619181 -0.00014161278134581956 -2.147771713810523e-5; 5.448663594687089e-5 0.0001311470748823658 -8.892957177342926e-5 -2.5869714789234542e-5 -2.209572104949373e-5 -4.35691383409085e-5 -6.181564296459689e-5 1.2768403195852853e-5 -1.876950793123668e-5 -0.00012170612373486367 -3.5345162523505233e-6 -5.5335202176160394e-5 1.7946633108292026e-5 2.211033805176252e-5 8.94875172725477e-5 8.134612815590486e-5 -4.6069757643661644e-5 4.93154455234372e-5 -1.317661573741949e-5 -0.00017563077037086465 5.2908222443848866e-5 8.844086352812544e-6 0.00019758804936097815 6.859305520444754e-5 -0.0001151134513002179 -4.8694002742215104e-5 2.9815355248476444e-5 -4.7955027383086356e-5 -8.991149135729442e-5 -4.0036417174650484e-5 -0.00018729621135028237 -2.638983298132043e-5; 0.000133919340595415 3.2755902947214286e-5 -5.660201698559724e-5 7.86548190887951e-5 0.00013219472036212122 -0.0001172243863322857 -9.038613628148657e-5 4.7424597295287474e-5 -5.8308214494205625e-5 3.6968143984961455e-5 0.00012996548336446546 -0.00013932243730891103 5.18684648056835e-5 0.00011294968878134037 -9.110456433630731e-5 -0.00010449621170777277 -0.0001667860195368453 -7.580587392225227e-5 -7.442677436414291e-5 -2.7451544002581004e-5 -7.379092115228174e-5 -5.159548512095821e-5 -2.0299670569546332e-5 0.00010111105090687857 0.00010861187199111696 0.0002037428350092863 2.1446883846477436e-5 0.0001156770814930091 0.0002626203351326348 5.821928440923739e-5 -2.126322199841341e-5 -7.17376313616445e-5; 9.550147860906143e-6 -2.9206162598673974e-5 -0.0002211517816067109 -6.046989381498061e-5 -4.212422880296043e-5 -0.00010656413667670701 -9.648753200959286e-5 -8.730928982909782e-5 1.9157758852126468e-5 0.00010323372100178152 0.00012881601707708272 5.91604699548439e-5 0.00015780546948012915 -0.00011936102544492182 -1.812765311111269e-5 -2.259498050950432e-6 -1.3990847382918405e-5 3.0323959999843624e-5 -2.6322396824942663e-5 -5.294015299983214e-6 0.00011406455871114435 2.8556686284029713e-5 5.1560517586078876e-5 -2.853621788741154e-5 8.319425956389387e-7 -0.00018346614831894425 -2.3135136307208164e-5 2.4379706355871727e-5 0.00014296692517749995 -0.00010418740146629185 8.852848277494969e-6 0.00011036459602113767; 1.821886338002198e-5 -3.7285701070285536e-7 -8.512751798550762e-5 7.08227308961596e-5 5.392289159482874e-6 -2.384246607861244e-6 8.334776078410641e-5 2.2337277306386997e-5 0.00010175445652860882 9.270622115875853e-5 -2.5859704572928295e-5 0.00013861719356211163 0.00015520831232711152 0.00013562056131489423 -8.521146691026305e-6 2.1673011480037648e-5 0.00017368351121261344 3.3054391798238884e-5 0.00011464856664245798 -1.9862240031932973e-5 0.00010191917693303769 -3.9929553048541896e-5 0.0001767611830757461 8.036903466857781e-5 -3.875432946079792e-5 4.772078801771112e-6 0.00011005491990657068 -3.934085531800149e-5 9.418335258693473e-6 -4.000176328991e-5 6.823767046737588e-5 -2.0561443187732683e-5; -0.00018025200494881278 9.847772946964647e-5 -0.00017573506135808918 6.149377498226134e-5 2.808775779011258e-6 -1.1262748895355843e-5 0.00012722820304484237 -6.447444263462427e-5 -1.5771483386874875e-5 0.0001430816852269833 9.42711200230848e-5 6.468648701124974e-5 8.658773788631836e-5 0.0001292380408166132 -2.193021043344321e-5 0.0001272145970041058 -5.945799363975794e-5 0.00010812952007550234 -2.2161289390300694e-5 2.8202932870816025e-5 1.647445845160797e-5 3.414195146329255e-5 -5.559699217987824e-5 -2.8276664462414054e-5 -6.95867341811742e-5 9.477231708737949e-5 0.00016174848918022183 -2.733235066548724e-5 -3.2178271785335697e-7 4.746364777824477e-5 -9.861513478531101e-5 -0.00014600231167709817; -0.00010365113498320617 1.2418538839276931e-5 9.82738891741758e-5 -5.350907681186889e-5 0.0001377125189299384 4.78364962453209e-5 0.00017362569983904753 -0.000162335036426362 -0.00011845781059148672 0.00010491978523875766 0.00011247942516436498 3.823613938382375e-5 -3.914383355204276e-5 -7.863384612048964e-5 -0.00014317383115324106 5.247332215165095e-5 0.00016348892300245432 1.4011924444071286e-5 -0.00012550047190072936 7.89658405244883e-5 -0.0001172963640294503 0.00022081846314338184 -1.9226364622446895e-6 5.9659494448872524e-5 -0.00025269751144716156 -0.00016572269318768982 2.8097830958153285e-5 -9.601175539698815e-6 -9.909885206636726e-5 -7.443099681237356e-6 1.756592778635907e-5 -0.0003146138443719123; 4.7710147744139536e-5 -3.0692046304832924e-5 -0.0001131230654140422 -4.2372890486521876e-5 -1.60705165330086e-5 -6.326708334419045e-5 5.684229673315238e-5 1.2908500327827094e-5 2.648390382782655e-5 -3.089733744891775e-5 5.8485662709785425e-5 -2.7402262087348514e-5 6.796149883265264e-5 7.512893905729244e-6 0.00010614756520917257 2.916774083823436e-5 -0.0001449519893798456 -0.00014660598916853274 -2.062477669133227e-5 0.0001789044637986027 2.2888087376119783e-5 1.6941725523425893e-5 -5.4859015829064734e-5 9.99041678438486e-5 -0.00015841936491816193 0.0001362693603437892 -7.860505665339919e-5 5.851977239907975e-5 0.00021300919089080328 5.05154368580341e-5 -7.793427156903282e-5 -2.4478291190829135e-5; -8.662054817912392e-6 -4.9234831742727036e-5 5.297914140762755e-5 -1.0049373479641875e-5 5.1571763840247934e-5 -3.183369516839852e-5 -0.00014209788241053073 2.6874329712223826e-5 3.237489546088595e-5 2.375660915938717e-5 2.3878945291720982e-5 0.0001593289907825918 4.957969759495103e-5 -0.00011273481685995948 -5.4426296622044544e-5 4.943721979295833e-5 -0.0001356413431428551 0.00015542380970807781 -5.375920772414547e-5 0.00013734547065475815 -8.262699156395981e-5 -4.043574335148501e-5 -5.650880665846161e-5 5.360943942585132e-5 -8.578752203241355e-5 -3.7613173838259265e-5 9.806889830295067e-5 -3.076581014531884e-5 -3.191347604363206e-5 0.00024313044378438707 8.727748900789793e-5 -0.00011839722900979133; 7.708134743049864e-7 -1.2819124864599608e-6 3.810765957982373e-5 -0.00015184038164502985 -5.193674331590568e-5 -0.00012245737269464015 5.4232262132559716e-5 0.0001613333650451258 0.00013219725844041566 2.8858769669883102e-5 -0.00013345946461836743 7.829163843357081e-5 0.00012829001099393564 -5.774376273641745e-8 9.664855935202097e-5 -2.5424262136351156e-6 8.874006636264852e-5 -7.28062670392812e-5 -9.865051941656382e-5 -5.57736649940623e-5 9.632654729589015e-5 -9.465232977586687e-5 5.3162179770284376e-5 -5.8901719225699705e-5 -0.00020074813181477608 -3.8354065677280935e-5 3.2756854866734926e-5 1.0960613340416789e-5 9.677575764303198e-5 -0.00013381424031163545 0.00016542488339413994 1.5369639098443778e-6; 0.00026350548910572704 0.00013037950453015764 7.573340304095174e-5 -0.00013175165326671705 -3.186099160799478e-5 -5.546180068038052e-5 -4.817747935288281e-5 0.000272154041571937 0.0001789205173323177 1.1968443273660835e-6 0.00013493710617200433 -0.00012556547403203892 -1.1960091565463886e-5 6.464877993001369e-5 -0.00014884458027647317 9.708422044657073e-5 7.479245437441902e-5 -3.588008871286021e-5 -7.161885400402593e-5 2.2474368283439805e-5 -0.00013396809913085052 0.00016674082622064612 -5.177302477516253e-5 -2.4336038635907072e-6 0.00015312352229970554 -7.475525739704492e-5 0.00018474531430418104 -4.8362124967247375e-5 0.0001494519285684644 7.156815741341479e-5 3.761965604514923e-5 8.204923256433186e-5; -4.969552118812647e-5 -0.00014198769658356475 -1.1567463293206563e-5 -4.8360357500060437e-5 1.582842979157848e-5 -0.00015769958863049018 -4.07812543878647e-5 -3.977338510117681e-5 -3.4437117751403127e-5 6.115520646312151e-5 -7.076194323849014e-5 -3.707879968237535e-5 4.3170814574568884e-5 -7.322012551845536e-5 -0.00013634299411456423 3.2368351232863485e-6 0.00016164403914921584 -0.00011728355766058952 1.536906402656322e-5 3.861183050071827e-5 -8.371072489786463e-5 4.002231490198331e-5 -6.278026139148423e-5 -0.00016161485418229822 8.223687892065795e-5 -2.7151734134116418e-5 0.00012717794434615135 4.989426507132621e-5 -0.00013978671940526898 -1.3817159374172322e-5 -3.7194563806001225e-5 -8.611200918927225e-5; 9.619146923552985e-5 -7.640272913949094e-5 5.715677086585045e-6 4.375983455004531e-5 -0.00018580714921215307 2.1232702163007306e-5 1.135082758748038e-5 2.2309869488991507e-5 3.085777909046555e-5 0.0001457458992420575 2.9449628685295126e-6 0.00019010258157988122 0.00011392867501612183 -0.00011875958542322263 3.3459748844789325e-5 -0.00011120476945751489 3.154520065191753e-5 -8.693382649897839e-5 -0.0002388003495822799 3.365969943599087e-5 -0.00011832026310247831 -0.0001133549895554905 -0.0002399014202480315 8.549270130342573e-6 -0.00013630408019041674 5.565274635206122e-7 4.0503308734797156e-5 -4.417170301410818e-5 5.806129063159741e-6 4.1772101137504234e-5 3.69033831860289e-5 9.268658952106326e-5; -7.149572278299232e-5 -0.00012945934080212814 -0.00013669299599874105 -1.3422268726705115e-5 -1.20720829146622e-5 -0.00012070877941018377 -2.4562637337594495e-6 7.879354003657228e-5 0.00022498333188545474 -5.87471754771785e-5 3.0587468371301987e-6 -0.00017994744351758812 7.272079298114344e-5 5.1818956729988904e-5 -0.00011399203734632042 9.175377405321878e-5 2.579993039384554e-5 -5.01366726313378e-5 -1.0377519662284423e-5 9.28505228001522e-5 3.066937973784812e-6 -6.197874111981491e-5 9.248840309597396e-6 -0.00012419704811341556 -0.00011588041755762656 0.00011968346140188088 9.138014635377183e-5 1.8153964869133437e-5 4.81143883588134e-5 -6.151743813111375e-5 9.677544354767218e-5 2.535289207884963e-6; 3.05768469098015e-5 -0.00019073152114464385 4.670459995218899e-5 4.617700571366198e-5 7.126801457893708e-5 -0.00017414219226516352 1.864441830371751e-5 1.4235466294903028e-5 6.301892673769526e-5 -2.013535166219387e-5 -0.00012290390921584156 0.00013266196285081352 1.2657197401535869e-5 6.477637609174056e-5 -6.610864264252124e-5 -0.00017350558962965965 -9.588512082877095e-5 -0.0001656416764325888 -0.00014114055855492548 1.7437386048237454e-5 0.00014731901566670792 3.633976550014196e-5 -0.00013743179923216445 0.0001284541455952967 -0.00011816405211170391 2.5964577326302726e-6 0.0002323479081661021 0.00017075020397719958 4.7513591851437573e-5 -1.887610239415017e-6 0.00029555333705326 6.656434081201924e-5; 9.092576405707502e-5 -7.198307825835317e-6 -2.896410037903505e-5 5.898720560403639e-6 -1.3235120745624605e-5 2.275778298671353e-7 -6.38164934100341e-6 1.5978540388291727e-6 2.802155799616468e-5 -0.0001038172569267943 2.763481356412405e-5 2.0020860433922213e-5 3.802358955468268e-5 9.671081247250459e-5 1.5755855970548687e-5 1.2840834501894115e-5 -7.902732718759376e-5 -2.6515813401437452e-5 0.00015919608890288423 8.446723117701902e-5 0.00010128347534662961 0.00020206013763998413 -3.855251739407311e-5 -9.788471429435149e-6 -3.487504832704036e-5 8.115921704773021e-5 -7.866613409971073e-5 -6.053131542542047e-5 -8.282965532527627e-5 7.0544327583524184e-6 3.044950321525962e-5 -4.417269933417249e-6; 1.573744403692683e-5 -5.919982076186897e-5 -5.938188487497022e-6 -3.53826901936576e-5 8.624077300744567e-5 6.663975064846324e-5 -7.17963514852524e-5 -0.0001442382811455257 -3.61846862596665e-5 2.1954380563393732e-5 -2.398147025748687e-6 0.00026438900086766094 -3.466681236197324e-5 0.00014532734049686955 -0.00016021676427947535 4.564145702722399e-5 -1.3743157332946454e-5 7.139082000328105e-5 0.00016314570958463268 -7.255500103376736e-5 5.727369417406408e-5 -1.8365374935391925e-5 2.4807081636091667e-5 -0.0001292501577062739 -7.222605499002754e-5 -5.122009614745074e-5 1.3430122499544248e-6 -7.116189217345791e-6 -0.0001500002866031799 8.385304382506962e-5 -0.0003222694825034787 0.00015827669695688226; 0.00014062014938158544 -0.00024629600980494225 4.9815234292181315e-5 0.00011812560237072788 -2.134533920556547e-5 0.00014485388359813684 -1.4766134568488331e-5 -0.0001387139770080134 -8.373914047785089e-5 7.159252831537473e-5 -0.00013273030222567844 9.654667641793059e-5 -9.278232633307845e-5 0.00010100427002238073 0.00014597356616344372 -8.932871301600339e-5 -0.00018677732319280445 -1.4269361286426345e-5 -2.0165600886107806e-5 4.72937730088638e-5 -0.00014221967890568125 3.62543008528851e-5 -2.0420743253781886e-5 9.050397179147207e-5 -0.0001067042964271842 -6.516104490724539e-6 -0.00013926600391220203 7.726835733105683e-5 8.491461753204755e-5 -0.00011352450988429214 -5.577752258919846e-5 -0.00018441627494700217], bias = [5.955308181677448e-10, 2.2930794526937992e-9, -2.0942252308576318e-9, -2.464863017902679e-10, -4.204560398559113e-9, 1.006675212028738e-9, 1.8826207645025094e-10, 1.288731938141571e-9, -1.8782556358586716e-9, -1.027687138710364e-9, 4.044580184373108e-10, -7.39385531640732e-10, 2.8518533042157587e-9, 1.7134991555940128e-10, -1.5895524768438176e-9, -1.4790366813349613e-9, 1.4553450130303518e-9, -6.506342820590809e-10, 4.918666704204747e-9, 1.608558573536849e-9, -1.6117576043599978e-9, 5.840844067111896e-10, 5.94816153106625e-10, 1.6626958861639982e-10, 4.326697837296909e-9, -3.480995255888671e-9, -1.7035587830137784e-9, -8.172138730822724e-10, 8.8286293256059e-10, 1.5851084600363675e-9, -6.723802743967432e-10, -2.2252200921226254e-9]), layer_4 = (weight = [-0.0005916092003726608 -0.0007735152217088616 -0.0005898637830574923 -0.0009455911375423543 -0.0006777394696437455 -0.0006431151099359726 -0.0007299059576953742 -0.0005452740966370144 -0.0005211313618496973 -0.0006389172842585809 -0.0007454850638026039 -0.0006928048311493951 -0.0007292858640763894 -0.0007932471961033723 -0.0006879893739434215 -0.000812883769508266 -0.0006047385586086192 -0.0007570487222109556 -0.0008703611817241053 -0.0007984241763479957 -0.0006075011400434991 -0.0006695837434432277 -0.0007022429771614517 -0.0008658297336685528 -0.0007076333317399777 -0.000560464979458682 -0.0006085106779573319 -0.0008117771218523721 -0.0006667696958789108 -0.0006304513413327485 -0.0005706061424896097 -0.000666481132504657; 0.0004523928818991084 0.0003163231458010248 0.00017001327546133056 7.104254173680311e-5 0.0001664434153889902 0.0001337161920058152 0.00023312026650055228 0.00011662115729559935 0.00029091188243712183 0.00041698749845960997 0.0003153319664483908 0.00015730328609884906 0.00023880371320333246 0.00027042886089394773 0.00013978959531510873 0.0002948130852679819 5.5414131697874084e-5 7.681315880077055e-5 0.0003030069744436119 0.0002452518552211384 0.0001608172437630492 0.0003053162745952673 0.0001470569640997154 0.0002785892366302851 0.0002539241384559333 0.0002856135817231763 0.0002812130280765807 0.00047863225193887666 6.754469186517595e-5 0.00022356898250117647 0.0003066188448046056 0.00020876783725708563], bias = [-0.000674450308603914, 0.00023315582560105969]))Visualizing the Results
Let us now plot the loss over time
julia
begin
fig = Figure()
ax = CairoMakie.Axis(fig[1, 1]; xlabel="Iteration", ylabel="Loss")
lines!(ax, losses; linewidth=4, alpha=0.75)
scatter!(ax, 1:length(losses), losses; marker=:circle, markersize=12, strokewidth=2)
fig
end
Finally let us visualize the results
julia
prob_nn = ODEProblem(ODE_model, u0, tspan, res.u)
soln_nn = Array(solve(prob_nn, RK4(); u0, p=res.u, saveat=tsteps, dt, adaptive=false))
waveform_nn_trained = first(compute_waveform(
dt_data, soln_nn, mass_ratio, ode_model_params))
begin
fig = Figure()
ax = CairoMakie.Axis(fig[1, 1]; xlabel="Time", ylabel="Waveform")
l1 = lines!(ax, tsteps, waveform; linewidth=2, alpha=0.75)
s1 = scatter!(
ax, tsteps, waveform; marker=:circle, alpha=0.5, strokewidth=2, markersize=12)
l2 = lines!(ax, tsteps, waveform_nn; linewidth=2, alpha=0.75)
s2 = scatter!(
ax, tsteps, waveform_nn; marker=:circle, alpha=0.5, strokewidth=2, markersize=12)
l3 = lines!(ax, tsteps, waveform_nn_trained; linewidth=2, alpha=0.75)
s3 = scatter!(ax, tsteps, waveform_nn_trained; marker=:circle,
alpha=0.5, strokewidth=2, markersize=12)
axislegend(ax, [[l1, s1], [l2, s2], [l3, s3]],
["Waveform Data", "Waveform Neural Net (Untrained)", "Waveform Neural Net"];
position=:lb)
fig
end
Appendix
julia
using InteractiveUtils
InteractiveUtils.versioninfo()
if @isdefined(MLDataDevices)
if @isdefined(CUDA) && MLDataDevices.functional(CUDADevice)
println()
CUDA.versioninfo()
end
if @isdefined(AMDGPU) && MLDataDevices.functional(AMDGPUDevice)
println()
AMDGPU.versioninfo()
end
endJulia Version 1.11.2
Commit 5e9a32e7af2 (2024-12-01 20:02 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 128 × AMD EPYC 7502 32-Core Processor
WORD_SIZE: 64
LLVM: libLLVM-16.0.6 (ORCJIT, znver2)
Threads: 16 default, 0 interactive, 8 GC (on 16 virtual cores)
Environment:
JULIA_CPU_THREADS = 16
JULIA_DEPOT_PATH = /cache/julia-buildkite-plugin/depots/01872db4-8c79-43af-ab7d-12abac4f24f6
JULIA_PKG_SERVER =
JULIA_NUM_THREADS = 16
JULIA_CUDA_HARD_MEMORY_LIMIT = 100%
JULIA_PKG_PRECOMPILE_AUTO = 0
JULIA_DEBUG = LiterateThis page was generated using Literate.jl.