Repository navigation
Tags: microsoft/SynapseML
Tags
chore: Upgrade to Spark 4.1.1, Java 17, Scala 2.13.17, Python 3.13 Spark 4.1 upgrade commit (one commit ahead of spark4.0). Key changes: - Spark 4.1.1, Scala 2.13.17 - Python 3.13 with updated dependencies - langchain 0.3.0 ecosystem (langchain-community, langchain-openai) - openai >= 1.58.1 (new API style) - LangchainTransformer rewritten to reconstruct LLM on workers (avoids pickling httpx clients with thread locks) - PEP 440 wheel filename normalization fix - Spark 4.x API compatibility fixes
chore: Upgrade to Spark 4.1.1, Java 17, Scala 2.13.17, Python 3.13 Spark 4.1 upgrade commit (one commit ahead of spark4.0). Key changes: - Spark 4.1.1, Scala 2.13.17 - Python 3.13 with updated dependencies - langchain 0.3.0 ecosystem (langchain-community, langchain-openai) - openai >= 1.58.1 (new API style) - LangchainTransformer rewritten to reconstruct LLM on workers (avoids pickling httpx clients with thread locks) - PEP 440 wheel filename normalization fix - Spark 4.x API compatibility fixes
[chore] Upgrade to Spark 4.0, Java 17, Scala 2.13, Python 3.12 Squashed from 3 previous upgrade commits: - Changing scala version to 2.13 for spark4.0 - Upgrade to Spark4.0, Java 17, Scala 2.13, Python 3.12.11 - Spark 4.0 release bug fixes (retired AzureMaps, Language API version, PEP 440, flash_attn, HasReturnUsage, Fuzzing.scala) Additional fixes during rebase: - Scala 2.13 collection compat (getAs[Seq] → scala.collection.Seq, null guards) - PySpark 4.0: .toDF() → createDataFrame() (removed API) - Spark 4.0 ANSI mode: try_element_at for array indexing - Pre-built cp312 wheels for horovod and pandas - Fabric E2E: auto-create workspace with Runtime 2.0, dynamic Scala suffix - DL module: try/except in __init__.py for safe jar initialization - Lazy transformers import in HuggingFaceCausalLMTransform
[chore] Upgrade to Spark 4.0, Java 17, Scala 2.13, Python 3.12 Squashed from 3 previous upgrade commits: - Changing scala version to 2.13 for spark4.0 - Upgrade to Spark4.0, Java 17, Scala 2.13, Python 3.12.11 - Spark 4.0 release bug fixes (retired AzureMaps, Language API version, PEP 440, flash_attn, HasReturnUsage, Fuzzing.scala) Additional fixes during rebase: - Scala 2.13 collection compat (getAs[Seq] → scala.collection.Seq, null guards) - PySpark 4.0: .toDF() → createDataFrame() (removed API) - Spark 4.0 ANSI mode: try_element_at for array indexing - Pre-built cp312 wheels for horovod and pandas - Fabric E2E: auto-create workspace with Runtime 2.0, dynamic Scala suffix - DL module: try/except in __init__.py for safe jar initialization - Lazy transformers import in HuggingFaceCausalLMTransform
fix: use checked-out tag for version instead of dynver (#2499) sbt-dynver uses git-describe to derive the version, which picks arbitrarily when multiple tags point to the same commit. This causes builds triggered from a specific tag (e.g. v1.1.1-spark4.1) to publish under a different tag's version (e.g. 1.1.1-python3.13). Fix: override dynver's version with the actual checked-out tag. - CI (ADO): uses BUILD_SOURCEBRANCH env var (refs/tags/v...) - Local: parses git reflog to find the tag from git checkout - Fallback: dynver (unchanged behavior for branch builds)
fix: use checked-out tag for version instead of dynver (#2499) sbt-dynver uses git-describe to derive the version, which picks arbitrarily when multiple tags point to the same commit. This causes builds triggered from a specific tag (e.g. v1.1.1-spark4.1) to publish under a different tag's version (e.g. 1.1.1-python3.13). Fix: override dynver's version with the actual checked-out tag. - CI (ADO): uses BUILD_SOURCEBRANCH env var (refs/tags/v...) - Local: parses git reflog to find the tag from git checkout - Fallback: dynver (unchanged behavior for branch builds)
fix: Master release bug fixes (#2485) * fix: Master release bug fixes - Handle retired Azure Maps Spatial API (CheckPointInPolygon throws on transform) - Remove AzMapsPointInPolygonSuite test class and add fuzzing exemptions - Update Language API version from 2022-10-01-preview to 2024-11-01 - Fix PEP 440 Python wheel version normalization (lowercase + local version) - Fix conda environment activation on Linux (use conda run) See: https://azure.microsoft.com/en-us/updates/v2/azure-maps-creator-services-retirement-on-30-september-2025 * test: Exclude geospatial notebooks from E2E tests Azure Maps Spatial API was retired on 9/30/2025. The notebooks 'Quickstart - Flooding Risk.ipynb' and 'Geospatial Services.ipynb' use CheckPointInPolygon which calls this retired API. Exclude these from DatabricksCPUTests to prevent E2E test failures.
PreviousNext