unity_sframe cannot be used alone in a pure C++ routine when pulling data from the remote system (S3). unity_sframe should be able to be used as a pure C++ library, which it's it is the original usage.

repro

#include <iostream>
#include <core/storage/fileio/s3_filesys.hpp>
#include <core/storage/fileio/general_fstream.hpp>
#include <core/storage/fileio/fixed_size_cache_manager.hpp>
#include <core/storage/serialization/dir_archive.hpp>
#include <core/storage/sframe_interface/unity_sframe.hpp>
#include <core/logging/logger.hpp>
#include <core/logging/assertions.hpp>
#include <core/globals/globals.hpp>

/**
 * Test the general_fstream.
 * This program will read from the file_url, writes "hello world" to the stream,
 * read from it and check contents are equal.
 */
int main(int argc, char** argv) {
  std::string url_read(
      "s3://"
// credentials
      "integration/manual/sframes/big_sframe_od/");

  global_logger().set_log_level(LOG_DEBUG);
  __attribute__((unused)) auto& gui = turi::fileio::s3::turi_global_AWS_SDK_setup();
  turi::globals::initialize_globals_from_environment(".");
  try {
    turi::unity_sframe sf;
    sf.construct_from_sframe_index(url_read);

    sf.begin_iterator();
    auto ret = sf.iterator_get_next(1);
    std::cout << ret.size() << std::endl;
    sf.save_frame("./gui");

  } catch(std::string& e) {
    std::cerr << "Exception: " << e << std::endl;
  } catch(std::exception& e) {
    std::cerr << "Exception: " << e.what() << std::endl;
  }

  return 0;
}

The root cause is that the file_handle singleton is destructed before block_cahe,

1584239579 : ERROR:    mutex.hpp(operator():68): Check failed: !error:
ERROR:    mutex.hpp(operator():68): Mutex lock error 22
1584239579 : ERROR:    mutex.hpp(operator():68): Check failed: !error:
ERROR:    mutex.hpp(operator():68): Mutex lock error 22
1584239579 : ERROR:    mutex.hpp(operator():68): Check failed: !error:
ERROR:    mutex.hpp(operator():68): Mutex lock error 22
1584239579 : ERROR:    mutex.hpp(operator():68): Check failed: !error:
ERROR:    mutex.hpp(operator():68): Mutex lock error 22
1584239579 : ERROR:    mutex.hpp(operator():68): Check failed: !error:
ERROR:    mutex.hpp(operator():68): Mutex lock error 22

Possible Solution

Similar to #3042, use dependency injection to avoid this error.

Further, we should have something to this

auto api = turi::initAPI(options);
...
turi::shutDown(api);