Using the starter code provided on the docker image, running the examples p_7 and s_7 result in the process being killed (n_9 runs fine).
To be precise, the process gets killed loading the data. I assume it's due to memory constraints but I haven't debugged the issue.
Is this issue local to me or is anyone else experiencing this issue?
Example output (the '...>>> get data' print is added in ingestion.py#L152 and there would have been a log call after the method, too):
root@9209dea230af:/app/codalab# python run_local_test.py --dataset_dir=./sample_data/p_7
2019-10-25 21:11:26,843 INFO ingestion.py: ===== Start ingestion program.
2019-10-25 21:11:26,844 INFO ingestion.py: ===== Initialize args.
2019-10-25 21:11:26,855 INFO ingestion.py: ===== Load metadata.
2019-10-25 21:11:26,855 INFO ingestion.py: Time budget: 1000
2019-10-25 21:11:26,855 INFO ingestion.py: ===== Set alive_thd
2019-10-25 21:11:26,856 INFO ingestion.py: ===== alive_thd started
2019-10-25 21:11:26,857 INFO ingestion.py: ===== Load user model
2019-10-25 21:11:27,960 INFO score.py: ===== init scoring program
2019-10-25 21:11:27,970 INFO score.py: ===== wait for ingestion to start
2019-10-25 21:11:27,981 INFO score.py: ===== detect alive ingestion
2019-10-25 21:11:27,981 INFO score.py: ===== wait for the exit of ingestion or end.txt file
Info:
{'task': 'pu', 'time_budget': 1000, 'schema': {'c_1': 'cat', 'c_2': 'cat', 'n_1': 'num', 'n_2': 'num', 'c_3': 'cat', 'c_4': 'cat', 'c_5': 'cat', 'n_3': 'num', 'c_6': 'cat', 'n_4': 'num', 'n_5': 'num', 'n_6': 'num', 'n_7': 'num', 'n_8': 'num', 'n_9': 'num', 'n_10': 'num', 'n_11': 'num', 'n_12': 'num', 'n_13': 'num', 'n_14': 'num', 'n_15': 'num', 'n_16': 'num', 'n_17': 'num', 'n_18': 'num', 'n_19': 'num', 'n_20': 'num', 'n_21': 'num', 'n_22': 'num', 'n_23': 'num', 'c_7': 'cat', 'c_8': 'cat', 'n_24': 'num', 'n_25': 'num', 'c_9': 'cat', 'c_10': 'cat', 'c_11': 'cat', 'n_26': 'num', 'c_12': 'cat', 'n_27': 'num', 'n_28': 'num', 'n_29': 'num', 'n_30': 'num', 'n_31': 'num', 'n_32': 'num', 'n_33': 'num', 'n_34': 'num', 'n_35': 'num', 'n_36': 'num', 'n_37': 'num', 'n_38': 'num', 'n_39': 'num', 'n_40': 'num', 'n_41': 'num', 'n_42': 'num', 'n_43': 'num', 'n_44': 'num', 'n_45': 'num', 'c_13': 'cat', 'n_46': 'num', 'n_47': 'num', 'n_48': 'num', 'n_49': 'num', 'n_50': 'num', 'n_51': 'num', 'n_52': 'num', 'n_53': 'num', 't_1': 'time', 'c_14': 'cat', 'c_15': 'cat'}, 'pred_time_budget': 1000}
2019-10-25 21:11:28,014 INFO ingestion.py: ===== Check user model methods
2019-10-25 21:11:28,014 INFO ingestion.py: ===== Begin training user model
2019-10-25 21:11:28,014 INFO ingestion.py: >>>>>>>>>>> get data
Killed
Traceback (most recent call last):
File "/app/codalab/scoring/score.py", line 319, in <module>
main()
File "/app/codalab/scoring/score.py", line 307, in main
raise RuntimeError("no end.txt exist, ingestion failed")
RuntimeError: no end.txt exist, ingestion failed