Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose Encoder and Decoder in TiktokenTokenizer #7314

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Fixing TiktokenTests to use the public properties Encoder and Decoder…

140aa42
Select commit
Loading
Failed to load commit list.
Draft

Expose Encoder and Decoder in TiktokenTokenizer #7314

Fixing TiktokenTests to use the public properties Encoder and Decoder…
140aa42
Select commit
Loading
Failed to load commit list.
Build Analysis / Build Analysis failed Jan 20, 2025 in 0s

.NET Result Analysis

Details

Tip

To unconditionally bypass the build analysis check (turn it green), you can use the escape mechanism feature. The completion time may vary, potentially taking several minutes, depending on the build analysis workload at the moment.

Build Failures

MachineLearning-CI / Windows_x64_Net6 / Windows_x64_Net6 Debug_Build / Run Helix Tests

[ 🚧 Report infrastructure issue] [ 📄 Report repository issue]
    ❌Cmd.exe exited with code '1'.

MachineLearning-CI / Windows_x64_Net6 / Windows_x64_Net6 Release_Build / Run Helix Tests

[ 🚧 Report infrastructure issue] [ 📄 Report repository issue]
    ❌Cmd.exe exited with code '1'.

Test Failures (4 tests failed)

🔹 [All failing tests from MachineLearning-CI]

    ❌ Microsoft.ML.OnnxTransformerTest.WorkItemExecution [Console] [Details] [Artifacts] [0.10% failure rate]
    [ 🚧 Report test infrastructure issue] [ 📄 Report test repository issue]
    This is a helix work item crash with status: BadExit. To investigate look the [Console log] / navigate to [Helix Artifacts]

    Failing Configuration
    ❌ Microsoft.ML.Tests.DnnImageFeaturizerTests.TestDnnImageFeaturizer [Console] [Details] [Artifacts] [0.10% failure rate]
    [ 🚧 Report test infrastructure issue] [ 📄 Report test repository issue]

    Failing Configuration
    Exception Message
    System.InvalidOperationException : Error during class instantiation
---- System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
-------- System.InvalidOperationException : Error during class instantiation
------------ System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
---------------- System.InvalidOperationException : Error initializing model :Microsoft.ML.OnnxRuntime.OnnxRuntimeException: [ErrorCode:Fail] inference_session.cc:1357 onnxruntime::InferenceSession::LoadOrtModelWithLoader ORT model verification failed.
   at Microsoft.ML.OnnxRuntime.InferenceSession.Init(String modelPath, SessionOptions options, PrePackedWeightsContainer prepackedWeightsContainer)
   at Microsoft.ML.OnnxRuntime.InferenceSession..ctor(String modelPath, SessionOptions options)
   at Microsoft.ML.Transforms.Onnx.OnnxModel..ctor(IHostEnvironment env, String modelFile, Nullable`1 gpuDeviceId, Boolean fallbackToCpu, Boolean ownModelFile, IDictionary`2 shapeDictionary, Int32 recursionLimit, Nullable`1 interOpNumThreads, Nullable`1 intraOpNumThreads) in D:\a\_work\1\s\src\Microsoft.ML.OnnxTransformer\OnnxUtils.cs:line 242
   at Microsoft.ML.Transforms.Onnx.OnnxTransformer..ctor(IHostEnvironment env, Options options, Byte[] modelBytes) in D:\a\_work\1\s\src\Microsoft.ML.OnnxTransformer\OnnxTransform.cs:line 280
-------------------- Microsoft.ML.OnnxRuntime.OnnxRuntimeException : [ErrorCode:Fail] inference_session.cc:1357 onnxruntime::InferenceSession::LoadOrtModelWithLoader ORT model verification failed.
    CallStack
       at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs) in D:\a\_work\1\s\src\Microsoft.ML.Core\ComponentModel\ComponentCatalog.cs:line 254
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra) in D:\a\_work\1\s\src\Microsoft.ML.Core\ComponentModel\ComponentCatalog.cs:line 1034
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra) in D:\a\_work\1\s\src\Microsoft.ML.Core\ComponentModel\ComponentCatalog.cs:line 993
   at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra) in D:\a\_work\1\s\src\Microsoft.ML.Core\Data\ModelLoading.cs:line 246
   at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra) in D:\a\_work\1\s\src\Microsoft.ML.Core\Data\ModelLoading.cs:line 172
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra) in D:\a\_work\1\s\src\Microsoft.ML.Core\Data\ModelLoading.cs:line 183
   at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra) in D:\a\_work\1\s\src\Microsoft.ML.Core\Data\ModelLoading.cs:line 111
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra) in D:\a\_work\1\s\src\Microsoft.ML.Core\Data\ModelLoading.cs:line 121
   at Microsoft.ML.ModelOperationsCatalog.Load(Stream stream, DataViewSchema& inputSchema) in D:\a\_work\1\s\src\Microsoft.ML.Data\Model\ModelOperationsCatalog.cs:line 163
   at Microsoft.ML.RunTests.TestDataPipeBase.TestEstimatorCore(IEstimator`1 estimator, IDataView validFitInput, IDataView validTransformInput, IDataView invalidInput, IDataView validForFitNotValidForTransformInput, Boolean shouldDispose) in D:\a\_work\1\s\test\Microsoft.ML.TestFramework\DataPipe\TestDataPipeBase.cs:line 88
   at Microsoft.ML.Tests.DnnImageFeaturizerTests.TestDnnImageFeaturizer() in D:\a\_work\1\s\test\Microsoft.ML.OnnxTransformerTest\DnnImageFeaturizerTest.cs:line 84
----- Inner Stack Trace -----
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs) in D:\a\_work\1\s\src\Microsoft.ML.Core\ComponentModel\ComponentCatalog.cs:line 254
----- Inner Stack Trace -----
   at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs) in D:\a\_work\1\s\src\Microsoft.ML.Core\ComponentModel\ComponentCatalog.cs:line 254
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra) in D:\a\_work\1\s\src\Microsoft.ML.Core\ComponentModel\ComponentCatalog.cs:line 1034
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra) in D:\a\_work\1\s\src\Microsoft.ML.Core\ComponentModel\ComponentCatalog.cs:line 993
   at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra) in D:\a\_work\1\s\src\Microsoft.ML.Core\Data\ModelLoading.cs:line 246
   at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra) in D:\a\_work\1\s\src\Microsoft.ML.Core\Data\Mod
    ❌ Microsoft.ML.Tests.OnnxConversionTest.NaiveCalibratorOnnxConversionTest [Console] [Details] [Artifacts] [0.19% failure rate]
    [ 🚧 Report test infrastructure issue] [ 📄 Report test repository issue]

    Failing Configuration
    Exception Message
    Assert.Equal() Failure: Values differ
Expected: 0
Actual:   1
    CallStack
       at Microsoft.ML.RunTests.BaseTestBaseline.Done() in D:\a\_work\1\s\test\Microsoft.ML.TestFramework\BaseTestBaseline.cs:line 221
   at Microsoft.ML.Tests.OnnxConversionTest.CommonCalibratorOnnxConversionTest(IEstimator`1 calibrator, IEstimator`1 calibratorNonStandard) in D:\a\_work\1\s\test\Microsoft.ML.Tests\OnnxConversionTest.cs:line 328
   at Microsoft.ML.Tests.OnnxConversionTest.NaiveCalibratorOnnxConversionTest() in D:\a\_work\1\s\test\Microsoft.ML.Tests\OnnxConversionTest.cs:line 349
    ❌ Microsoft.ML.Tests.WorkItemExecution [Console] [Details] [Artifacts] [0.92% failure rate]
    [ 🚧 Report test infrastructure issue] [ 📄 Report test repository issue]
    This is a helix work item crash with status: BadExit. To investigate look the [Console log] / navigate to [Helix Artifacts]

    Failing Configuration

Was this helpful? Yes No

View more details on Build Analysis