Overview
I'm experiencing a critical issue where TensorFlow-metal and PyArrow seem to be incompatible when installed together in the same environment. Whenever both packages are present, TensorFlow crashes and the kernel dies during execution. Environment Details
Environment Details
- macOS Version: 15.3.2
- Mac Model: MacBook Pro Max M3
- Python Version: 3.11
- TensorFlow Version: 2.19
- PyArrow Version: 19.0.0
Issue Description:
When both TensorFlow-metal and PyArrow are installed in the same Python environment, any attempt to use TensorFlow results in immediate kernel crashes. The issue appears to be a compatibility problem between these two packages rather than a problem with either package individually.
Steps to Reproduce
-
Create a new Python environment:
conda create -n tf-metal python=3.11 -
Install TensorFlow-metal:
pip install tensorflow tensorflow-metalInstall PyArrow: pip install pyarrow -
Run the following minimal example:
# Create a simple model
model = tf.keras.Sequential([
tf.keras.layers.Input(shape=(2,)),
tf.keras.layers.Dense(1)
])
model.compile(optimizer='adam', loss='mse')
model.summary() # This works fine
# Generate some dummy data
X = np.random.random((100, 2))
y = np.random.random((100, 1))
# The crash happens exactly at this line
model.fit(X, y, epochs=5, batch_size=32) # CRASH: Kernel dies here
- Result: Kernel crashes with no error message
What I've Tried
Reinstalling both packages in different orders Using different versions of both packages Creating isolated environments Checking system logs for additional error information
The only workaround I've found is to use separate environments for each package, which isn't practical for my workflow as I need both libraries for my data processing and machine learning pipeline.
Questions
Has anyone else encountered this specific compatibility issue? Are there known workarounds that allow both packages to coexist? Is this a known issue that's being addressed in upcoming releases?
Any insights, suggestions, or assistance would be greatly appreciated. I'm happy to provide any additional information that might help diagnose this problem. Thank you in advance for your help!
Thank you in advance for your help!
Hello,
Please take a moment to create a bug report with the Feedback Assistant.
Otherwise, have you considered MLX and whether it could meet your needs?
"MLX is an array framework optimized for the unified memory architecture of Apple silicon. The NumPy-like API makes it familiar to use and flexible. The higher level neural net and optimizer packages along with function transformations for automatic differentiation and graph optimization let you build more complex yet efficient machine learning models. MLX also has Swift, C++, and C bindings and can run on any Apple platform."
