Error when inserting large binary files using EF4
Error when inserting large binary files using EF4
Hello Devart,
I am experiencing problems when inserting binary files into a postgres bytea column using Entity Framework 4 and dotConnect. When inserting a file i get either a System.InvalidOperationException("Connection must be opened.") (150 MB file) or a System.OutOfMemoryException (200+ MB file).
I have a test project available demonstrating the behavior. Do you have a place where i may upload it?
Full tracebacks for the Exceptions are as follows:
System.InvalidOperationException
Message: "Connection must be opened."
StackTrace:
at Devart.Common.Utils.CheckConnectionOpen(IDbConnection connection)
at Devart.Data.PostgreSql.PgSqlConnection.Rollback()
at Devart.Data.PostgreSql.PgSqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Objects.ObjectContext.SaveChanges()
at LargeFileTest.Program.Main(String[] args) in C:\test\LargeFileTest\LargeFileTest\Program.cs:line 25
System.OutOfMemoryException:
Message: "Exception of type 'System.OutOfMemoryException' was thrown."
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Devart.Data.PostgreSql.o.a(Byte[] A_0, Int32 A_1, Int32 A_2, Int32 A_3)
at Devart.Data.PostgreSql.o.b(Byte[] A_0, Int32 A_1, Int32 A_2)
at Devart.Data.PostgreSql.o.a(Char A_0)
at Devart.Data.PostgreSql.s.b(Char A_0)
at Devart.Data.PostgreSql.s.c(String A_0)
at Devart.Data.PostgreSql.s.g(af A_0)
at Devart.Data.PostgreSql.s.e(af A_0)
at Devart.Data.PostgreSql.af.ah()
at Devart.Data.PostgreSql.PgSqlCommand.InternalPrepare(Boolean implicitPrepare, Int32 startRecord, Int32 maxRecords)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.ExecuteReader()
at Devart.Common.DbCommandBase.ExecuteNonQuery()
at Devart.Data.PostgreSql.PgSqlConnection.Rollback()
at Devart.Data.PostgreSql.PgSqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Objects.ObjectContext.SaveChanges()
at LargeFileTest.Program.Main(String[] args) in C:\test\LargeFileTest\LargeFileTest\Program.cs:line 25
Best regards,
Anders
I am experiencing problems when inserting binary files into a postgres bytea column using Entity Framework 4 and dotConnect. When inserting a file i get either a System.InvalidOperationException("Connection must be opened.") (150 MB file) or a System.OutOfMemoryException (200+ MB file).
I have a test project available demonstrating the behavior. Do you have a place where i may upload it?
Full tracebacks for the Exceptions are as follows:
System.InvalidOperationException
Message: "Connection must be opened."
StackTrace:
at Devart.Common.Utils.CheckConnectionOpen(IDbConnection connection)
at Devart.Data.PostgreSql.PgSqlConnection.Rollback()
at Devart.Data.PostgreSql.PgSqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Objects.ObjectContext.SaveChanges()
at LargeFileTest.Program.Main(String[] args) in C:\test\LargeFileTest\LargeFileTest\Program.cs:line 25
System.OutOfMemoryException:
Message: "Exception of type 'System.OutOfMemoryException' was thrown."
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Devart.Data.PostgreSql.o.a(Byte[] A_0, Int32 A_1, Int32 A_2, Int32 A_3)
at Devart.Data.PostgreSql.o.b(Byte[] A_0, Int32 A_1, Int32 A_2)
at Devart.Data.PostgreSql.o.a(Char A_0)
at Devart.Data.PostgreSql.s.b(Char A_0)
at Devart.Data.PostgreSql.s.c(String A_0)
at Devart.Data.PostgreSql.s.g(af A_0)
at Devart.Data.PostgreSql.s.e(af A_0)
at Devart.Data.PostgreSql.af.ah()
at Devart.Data.PostgreSql.PgSqlCommand.InternalPrepare(Boolean implicitPrepare, Int32 startRecord, Int32 maxRecords)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.ExecuteReader()
at Devart.Common.DbCommandBase.ExecuteNonQuery()
at Devart.Data.PostgreSql.PgSqlConnection.Rollback()
at Devart.Data.PostgreSql.PgSqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Objects.ObjectContext.SaveChanges()
at LargeFileTest.Program.Main(String[] args) in C:\test\LargeFileTest\LargeFileTest\Program.cs:line 25
Best regards,
Anders
The reasons of the issue.
1. The Entity Framework side.
Additional memory costs by Entity Framework application are caused by change tracking, numerous additional ORM structures in the memory, and usage of standard code in the setter of binary property:
The SetValidValue code clones array of bytes in its implementation, which leads to the additional memory costs. We do not know the reasons why Microsoft chose such behaviour of StructuralObject.SetValidValue for the byte array.
2. The ADO.NET side.
Currently, our ADO.NET level increases an amount of used memory in several times comparing to the size of actual bytea data. We are investigating the possibility to change the approach of handling such situation.
1. The Entity Framework side.
Additional memory costs by Entity Framework application are caused by change tracking, numerous additional ORM structures in the memory, and usage of standard code in the setter of binary property:
Code: Select all
_Byteacolumn = StructuralObject.SetValidValue(value, true);
2. The ADO.NET side.
Currently, our ADO.NET level increases an amount of used memory in several times comparing to the size of actual bytea data. We are investigating the possibility to change the approach of handling such situation.
Re: Error when inserting large binary files using EF4
I get the same error in Oracle. I imagine this problem extends to all the databases.
Re: Error when inserting large binary files using EF4
Thank you for your report. We are investigating the issue.curelom wrote:I get the same error in Oracle. I imagine this problem extends to all the databases.
Re: Error when inserting large binary files using EF4
I am having the same issue using PostgreSQL LinqConnect..
Re: Error when inserting large binary files using EF4
Thank you for the report. We have reproduced this issue with LinqConnect. We will investigate it and inform you about the results as soon as possible.muthu wrote:I am having the same issue using PostgreSQL LinqConnect..
Re: Error when inserting large binary files using EF4
any news on the bug fix ?
Re: Error when inserting large binary files using EF4
The investigation is in the progress. We will keep you updated concerning the status of the corresponding request.
Re: Error when inserting large binary files using EF4
I encounter a similar problem inserting large binary files into a mysql database through EF4(using devart's dotConnect for MySQL).
I get the exception and message below when inserting mp4 video files, approx. 4mb in size.
System.InvalidOperationException
Message: "Connection must be opened."
StackTrace:
at Devart.Common.Utils.CheckConnectionOpen(IDbConnection connection)
at Devart.Data.MySql.MySqlConnection.Rollback()
at Devart.Data.MySql.MySqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Entity.Internal.InternalContext.SaveChanges()
at System.Data.Entity.Internal.LazyInternalContext.SaveChanges()
at System.Data.Entity.DbContext.SaveChanges()
I've found that if I only import the binary objects for pdfs of similar size(~2.5mb) that everything works well.
Is this still an open problem for dotConnect for MySQL?
Thank you,
Keenan
I get the exception and message below when inserting mp4 video files, approx. 4mb in size.
System.InvalidOperationException
Message: "Connection must be opened."
StackTrace:
at Devart.Common.Utils.CheckConnectionOpen(IDbConnection connection)
at Devart.Data.MySql.MySqlConnection.Rollback()
at Devart.Data.MySql.MySqlTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.EntityClient.EntityTransaction.Dispose(Boolean disposing)
at System.Data.Common.DbTransaction.Dispose()
at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options)
at System.Data.Entity.Internal.InternalContext.SaveChanges()
at System.Data.Entity.Internal.LazyInternalContext.SaveChanges()
at System.Data.Entity.DbContext.SaveChanges()
I've found that if I only import the binary objects for pdfs of similar size(~2.5mb) that everything works well.
Is this still an open problem for dotConnect for MySQL?
Thank you,
Keenan
Re: Error when inserting large binary files using EF4
Does the issue persist with the latest (7.7.301) build of dotConnect for MySQL? Please send us a small test project with the corresponding DDL/DML script so that we can reproduce the issue in our environment.
Re: Error when inserting large binary files using EF4
The upload issue was resolved by increasing the max_allowed_packet to 50MB in mysql. Although when attempting to retrieve any file over 10MB I get the following error:
System.ArgumentException
Source: Devart.Data.MySql
Message: Cannot retrieve huge data in FetchAll mode
StackTrace:
at Devart.Data.MySql.bl.b(Boolean A_0)
at Devart.Data.MySql.bl.n()
at Devart.Data.MySql.bu.m()
at Devart.Data.MySql.af.b(ab[] A_0, Int32 A_1)
at Devart.Data.MySql.af.a(Byte[] A_0, Int32 A_1, Boolean A_2)
at Devart.Data.MySql.i.b()
at Devart.Data.MySql.i.v()
at Devart.Data.MySql.MySqlCommand.InternalExecute(CommandBehavior behavior, IDisposable stmt, Int32 startRecord, Int32 maxRecords)
at Devart.Common.DbCommandBase.InternalExecute(CommandBehavior behavior, IDisposable stmt, Int32 startRecord, Int32 maxRecords, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior)
at Devart.Data.MySql.Entity.x.a(CommandBehavior A_0)
at Devart.Common.Entity.eb.b(CommandBehavior A_0)
at System.Data.EntityClient.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
Here's the EF4 code that throws the error:
using (SalesModelDB db = new SalesModelDB())
{
fileData = db.DataDictionaries
.Where(d => d.Id == fileId)
.Select(f => f.FileData)
.FirstOrDefault();
}
Is there a way to increase the fetch limit or switch the mode from fetchall to something else to allow me to get these larger binary files?
Thank you,
Keenan
System.ArgumentException
Source: Devart.Data.MySql
Message: Cannot retrieve huge data in FetchAll mode
StackTrace:
at Devart.Data.MySql.bl.b(Boolean A_0)
at Devart.Data.MySql.bl.n()
at Devart.Data.MySql.bu.m()
at Devart.Data.MySql.af.b(ab[] A_0, Int32 A_1)
at Devart.Data.MySql.af.a(Byte[] A_0, Int32 A_1, Boolean A_2)
at Devart.Data.MySql.i.b()
at Devart.Data.MySql.i.v()
at Devart.Data.MySql.MySqlCommand.InternalExecute(CommandBehavior behavior, IDisposable stmt, Int32 startRecord, Int32 maxRecords)
at Devart.Common.DbCommandBase.InternalExecute(CommandBehavior behavior, IDisposable stmt, Int32 startRecord, Int32 maxRecords, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior, Boolean nonQuery)
at Devart.Common.DbCommandBase.ExecuteDbDataReader(CommandBehavior behavior)
at Devart.Data.MySql.Entity.x.a(CommandBehavior A_0)
at Devart.Common.Entity.eb.b(CommandBehavior A_0)
at System.Data.EntityClient.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
Here's the EF4 code that throws the error:
using (SalesModelDB db = new SalesModelDB())
{
fileData = db.DataDictionaries
.Where(d => d.Id == fileId)
.Select(f => f.FileData)
.FirstOrDefault();
}
Is there a way to increase the fetch limit or switch the mode from fetchall to something else to allow me to get these larger binary files?
Thank you,
Keenan
Re: Error when inserting large binary files using EF4
Thank you for the report. We have reproduced this issue. We will investigate it and inform you about the results as soon as possible.Keenan wrote: Although when attempting to retrieve any file over 10MB I get the following error:
System.ArgumentException
Source: Devart.Data.MySql
Message: Cannot retrieve huge data in FetchAll mode