Split Data Frame into Smaller Chunks

Aug 2, 2018 #mysql

When writing from Docker container to MySQL,

Error: MySQL server has gone away...

try split your dataset into smaller chunks,

library(tidyverse)
library(MASS)

data("Boston")
df <- Boston

# cut into desired chunks, here chunk size = 100
indices <- (nrow(df) %>% seq() - 1) %/% 100
df_chunks <- split(df, indices)

and write into database with iteration.

library(DBI)

# remember to append, do not overwrite
lapply(df_chunks, dbWriteTable, conn = mycon, name = mytable, append = TRUE, row.names = FALSE)