Commit 3d169f43 authored by Nathan Bronson's avatar Nathan Bronson Committed by Facebook Github Bot

memory savings for F14 tables with explicit reserve()

Summary:
In the case when an explicit capacity is specified (via reserve()
or an initial capacity) we can save memory by using a bucket_count()
off of the normal geometric sequence.  This is beneficial for sizes
<= Chunk::kCapacity for all policies, and for F14Vector tables with
any size.  In the multi-chunk F14Vector case this will save about
40%*size()*sizeof(value_type) when reserve() is used (such as during
Thrift deserialization).  The single-chunk savings are potentially larger.
The changes do not affect the lookup path and should be a tiny perf win in
the non-growing insert case.  Exact sizing is only attempted on reserve()
or rehash() when the requested capacity is >= 9/8 or <= 7/8 of the current
bucket_count(), so it won't trigger O(n^2) behavior even if misused.

Reviewed By: yfeldblum

Differential Revision: D12848355

fbshipit-source-id: 4f70b4dabf626142cfe370e5b1db581af1a1103f
parent 175e5c95
......@@ -206,7 +206,20 @@ are not used. That means that we can also support capacities that are
a fraction of 1 chunk with no change to any of the search and insertion
algorithms. The only change required is in the check to see if a rehash
is required. F14's first three capacities all use one chunk and one
16-byte metadata vector, but allocate space for 2, 6, and then 12 keys.
16-byte metadata vector, but allocate space for 2, 6, and then 14 keys.
## MEMORY OVERHEAD WITH EXPLICIT CAPACITY REQUEST
When using the vector storage strategy F14 has the option of sizing the
main hash array and the value_type storage array independently. In the
case that an explicit capacity has been requested (`initialCapacity`
passed to a constructor or a call to `reserve` or `rehash`) we take
advantage of this, trying to exactly fit the capacity to the requested
quantity. We also try to exactly fit the capacity for the other storage
strategies if there is only a single chunk. To avoid pathologies from
code that calls `reserve` a lot, doubling is performed for increases
if there is not at least a 1/8 increase in capacity and decreases are
ignored if there is not at least a 1/8 decrease.
## IS F14NODEMAP FULLY STANDARDS-COMPLIANT?
......
......@@ -465,7 +465,11 @@ class F14BasicMap {
FOLLY_ALWAYS_INLINE void
bulkInsert(InputIt first, InputIt last, bool autoReserve) {
if (autoReserve) {
table_.reserveForInsert(std::distance(first, last));
auto n = std::distance(first, last);
if (n == 0) {
return;
}
table_.reserveForInsert(n);
}
while (first != last) {
insert(*first);
......@@ -1155,6 +1159,11 @@ class F14VectorMapImpl : public F14BasicMap<MapPolicyWithDefaults<
// inherit constructors
using Super::Super;
F14VectorMapImpl& operator=(std::initializer_list<value_type> ilist) {
Super::operator=(ilist);
return *this;
}
iterator begin() {
return this->table_.linearBegin(this->size());
}
......
......@@ -404,7 +404,11 @@ class F14BasicSet {
FOLLY_ALWAYS_INLINE void
bulkInsert(InputIt first, InputIt last, bool autoReserve) {
if (autoReserve) {
table_.reserveForInsert(std::distance(first, last));
auto n = std::distance(first, last);
if (n == 0) {
return;
}
table_.reserveForInsert(n);
}
while (first != last) {
insert(*first);
......@@ -899,6 +903,11 @@ class F14VectorSetImpl : public F14BasicSet<SetPolicyWithDefaults<
using Super::Super;
F14VectorSetImpl& operator=(std::initializer_list<value_type> ilist) {
Super::operator=(ilist);
return *this;
}
iterator begin() {
return cbegin();
}
......
......@@ -197,6 +197,10 @@ struct BasePolicy
using MappedOrBool = std::conditional_t<kIsMap, Mapped, bool>;
// if true, bucket_count() after reserve(n) will be as close as possible
// to n for multi-chunk tables
static constexpr bool kContinuousCapacity = false;
//////// methods
BasePolicy(Hasher const& hasher, KeyEqual const& keyEqual, Alloc const& alloc)
......@@ -1020,6 +1024,8 @@ class VectorContainerPolicy : public BasePolicy<
public:
static constexpr bool kEnableItemIteration = false;
static constexpr bool kContinuousCapacity = true;
using InternalSizeType = Item;
using ConstIter =
......@@ -1318,11 +1324,11 @@ class VectorContainerPolicy : public BasePolicy<
private:
// Returns the byte offset of the first Value in a unified allocation
// that first holds prefixBytes of data, where prefixBytes comes from
// Chunk storage and hence must be at least 8-byte aligned (sub-Chunk
// allocations always have an even capacity and sizeof(Item) == 4).
// Chunk storage and may be only 4-byte aligned due to sub-chunk
// allocation.
static std::size_t valuesOffset(std::size_t prefixBytes) {
FOLLY_SAFE_DCHECK((prefixBytes % 8) == 0, "");
if (alignof(Value) > 8) {
FOLLY_SAFE_DCHECK((prefixBytes % alignof(Item)) == 0, "");
if (alignof(Value) > alignof(Item)) {
prefixBytes = -(-prefixBytes & ~(alignof(Value) - 1));
}
FOLLY_SAFE_DCHECK((prefixBytes % alignof(Value)) == 0, "");
......
This diff is collapsed.
......@@ -50,7 +50,6 @@ TEST(F14Map, customSwap) {
testCustomSwap<folly::F14FastMap>();
}
namespace {
template <
template <typename, typename, typename, typename, typename> class TMap,
typename K,
......@@ -116,7 +115,6 @@ void runAllocatedMemorySizeTests() {
runAllocatedMemorySizeTest<folly::F14VectorMap, K, V>();
runAllocatedMemorySizeTest<folly::F14FastMap, K, V>();
}
} // namespace
TEST(F14Map, getAllocatedMemorySize) {
runAllocatedMemorySizeTests<bool, bool>();
......@@ -128,6 +126,86 @@ TEST(F14Map, getAllocatedMemorySize) {
runAllocatedMemorySizeTests<folly::fbstring, long>();
}
template <typename M>
void runContinuousCapacityTest(std::size_t minSize, std::size_t maxSize) {
using K = typename M::key_type;
for (std::size_t n = minSize; n <= maxSize; ++n) {
M m1;
m1.reserve(n);
auto cap = m1.bucket_count();
double ratio = cap * 1.0 / n;
// worst case scenario is that rehash just occurred and capacityScale
// is 5*2^12
EXPECT_TRUE(ratio < 1 + 1.0 / (5 << 12))
<< ratio << ", " << cap << ", " << n;
m1.try_emplace(0);
M m2;
m2 = m1;
EXPECT_LE(m2.bucket_count(), 2);
for (K i = 1; i < n; ++i) {
m1.try_emplace(i);
}
EXPECT_EQ(m1.bucket_count(), cap);
M m3 = m1;
EXPECT_EQ(m3.bucket_count(), cap);
for (K i = n; i <= cap; ++i) {
m1.try_emplace(i);
}
EXPECT_GT(m1.bucket_count(), cap);
EXPECT_LE(m1.bucket_count(), 3 * cap);
M m4;
for (K i = 0; i < n; ++i) {
m4.try_emplace(i);
}
// reserve(0) works like shrink_to_fit. Note that tight fit (1/8
// waste bound) only applies for vector policy or single-chunk, which
// might not apply to m1. m3 should already have been optimally sized.
m1.reserve(0);
m3.reserve(0);
m4.reserve(0);
EXPECT_GT(m1.load_factor(), 0.5);
EXPECT_GE(m3.load_factor(), 0.875);
EXPECT_EQ(m3.bucket_count(), cap);
EXPECT_GE(m4.load_factor(), 0.875);
}
}
TEST(F14Map, continuousCapacitySmall) {
runContinuousCapacityTest<folly::F14NodeMap<std::size_t, std::string>>(1, 14);
runContinuousCapacityTest<folly::F14ValueMap<std::size_t, std::string>>(
1, 14);
runContinuousCapacityTest<folly::F14VectorMap<std::size_t, std::string>>(
1, 100);
runContinuousCapacityTest<folly::F14FastMap<std::size_t, std::string>>(
1, 100);
}
TEST(F14Map, continuousCapacityBig0) {
runContinuousCapacityTest<folly::F14VectorMap<std::size_t, std::string>>(
1000000 - 1, 1000000 - 1);
}
TEST(F14Map, continuousCapacityBig1) {
runContinuousCapacityTest<folly::F14VectorMap<std::size_t, std::string>>(
1000000, 1000000);
}
TEST(F14Map, continuousCapacityBig2) {
runContinuousCapacityTest<folly::F14VectorMap<std::size_t, std::string>>(
1000000 + 1, 1000000 + 1);
}
TEST(F14Map, continuousCapacityBig3) {
runContinuousCapacityTest<folly::F14VectorMap<std::size_t, std::string>>(
1000000 + 2, 1000000 + 2);
}
TEST(F14Map, continuousCapacityF12) {
runContinuousCapacityTest<folly::F14VectorMap<uint16_t, uint16_t>>(
0xfff0, 0xfffe);
}
template <typename M>
void runVisitContiguousRangesTest(int n) {
M map;
......@@ -208,6 +286,17 @@ void runSimple() {
T h;
EXPECT_EQ(h.size(), 0);
h.reserve(0);
std::vector<std::pair<std::string const, std::string>> v(
{{"abc", "first"}, {"abc", "second"}});
h.insert(v.begin(), v.begin());
EXPECT_EQ(h.size(), 0);
EXPECT_EQ(h.bucket_count(), 0);
h.insert(v.begin(), v.end());
EXPECT_EQ(h.size(), 1);
EXPECT_EQ(h["abc"], s("first"));
h = T{};
EXPECT_EQ(h.bucket_count(), 0);
h.insert(std::make_pair(s("abc"), s("ABC")));
EXPECT_TRUE(h.find(s("def")) == h.end());
......
......@@ -201,6 +201,15 @@ void runSimple() {
T h;
EXPECT_EQ(h.size(), 0);
h.reserve(0);
std::vector<std::string> v({"abc", "abc"});
h.insert(v.begin(), v.begin());
EXPECT_EQ(h.size(), 0);
EXPECT_EQ(h.bucket_count(), 0);
h.insert(v.begin(), v.end());
EXPECT_EQ(h.size(), 1);
h = T{};
EXPECT_EQ(h.bucket_count(), 0);
h.insert(s("abc"));
EXPECT_TRUE(h.find(s("def")) == h.end());
......
......@@ -17,6 +17,7 @@
#pragma once
#include <cstddef>
#include <cstring>
#include <limits>
#include <memory>
#include <ostream>
......@@ -394,12 +395,21 @@ class SwapTrackingAlloc {
++testAllocationCount;
testAllocatedMemorySize += n * sizeof(T);
++testAllocatedBlockCount;
return a_.allocate(n);
std::size_t extra =
std::max<std::size_t>(1, sizeof(std::size_t) / sizeof(T));
T* p = a_.allocate(extra + n);
std::memcpy(p, &n, sizeof(std::size_t));
return p + extra;
}
void deallocate(T* p, size_t n) {
testAllocatedMemorySize -= n * sizeof(T);
--testAllocatedBlockCount;
a_.deallocate(p, n);
std::size_t extra =
std::max<std::size_t>(1, sizeof(std::size_t) / sizeof(T));
std::size_t check;
std::memcpy(&check, p - extra, sizeof(std::size_t));
FOLLY_SAFE_CHECK(check == n, "");
a_.deallocate(p - extra, n + extra);
}
private:
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment